guix: parallelize LIEF build #25458
Merged
guix: parallelize LIEF build #25458
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Member
|
Concept ACK |
Contributor
Guix builds
|
Member
|
Not sure why this failed, maybe a cosmic ray? |
Contributor
|
Concept ACK, GUIX hashes: x86: arm64: |
Member
|
Build on x86_64, matches @jarolrod at least Code review ACK 8d25926 |
Member
I think so. I checked the |
Member
Guix builds on
|
Contributor
sidhujag
pushed a commit
to syscoin/syscoin
that referenced
this pull request
Jun 27, 2022
8d25926 guix: parallelize LIEF build (fanquake) Pull request description: Even though it's (re)built infrequently, the LIEF build currently runs with a single job, and thus takes much longer to build than it should. This change modifies our packge so that the cmake invocation is done with multiple jobs, using [`parallel-job-count`](https://git.savannah.gnu.org/cgit/guix.git/tree/build-aux/compile-all.scm#n49). On a 6 core arm64 machine, this results in a ~80% speed up, and reduces the time taken to build the LIEF packge from ~19m to ~4m. Guix Build (x86_64): ```bash ba12c164915ec3f5bd916c99f75e074b935028acc7b65914d109b19caf8f86e7 guix-build-8d25926643cb/output/aarch64-linux-gnu/SHA256SUMS.part 09d062446d8e8edd1c312da83d8fb1a4998880b028afe0bef40336bc0971358f guix-build-8d25926643cb/output/aarch64-linux-gnu/bitcoin-8d25926643cb-aarch64-linux-gnu-debug.tar.gz 74f1dad26cca571ee4eafba70537067be610059f770408460ae0216fd2fa4f7f guix-build-8d25926643cb/output/aarch64-linux-gnu/bitcoin-8d25926643cb-aarch64-linux-gnu.tar.gz 900e37dd5216f06b6e32af065336a6d1aff2fead13b3394fa521401eefa5b388 guix-build-8d25926643cb/output/arm-linux-gnueabihf/SHA256SUMS.part 4c8361225f9408cd44e08421ba39a13758bbba2159a4c3f1a5562e83175904af guix-build-8d25926643cb/output/arm-linux-gnueabihf/bitcoin-8d25926643cb-arm-linux-gnueabihf-debug.tar.gz 4780381b4e51ed99f6967975291703584e71dd2a22ea6bc77716d597f09bdbdd guix-build-8d25926643cb/output/arm-linux-gnueabihf/bitcoin-8d25926643cb-arm-linux-gnueabihf.tar.gz 46e178b38ca9a47ebcd4dd3d037f08c49bf79faf09c032e3a0ec99974bc47bcd guix-build-8d25926643cb/output/arm64-apple-darwin/SHA256SUMS.part e44a9fa027b912a13ceb12708eaf864f5fed896887de8ec157c9c3bba24f9d64 guix-build-8d25926643cb/output/arm64-apple-darwin/bitcoin-8d25926643cb-arm64-apple-darwin-unsigned.dmg 82bd6c2995f1c707b0204c618f7f16a90a80b5378a856cb287bad4cdf2191467 guix-build-8d25926643cb/output/arm64-apple-darwin/bitcoin-8d25926643cb-arm64-apple-darwin-unsigned.tar.gz 9b28de51d7adea26ffae22f9a4db517643297aeab24467efe4916088fda496d0 guix-build-8d25926643cb/output/arm64-apple-darwin/bitcoin-8d25926643cb-arm64-apple-darwin.tar.gz 71d86ffaf1b54b1dfb0ee6a978c2d3c800742d82a7845967becdd88b1ba4cbf4 guix-build-8d25926643cb/output/dist-archive/bitcoin-8d25926643cb.tar.gz 9c19a5dd32586fa1a3e82a1d7882ac248dc4bd7fe525c0282b245f04d9618e4b guix-build-8d25926643cb/output/powerpc64-linux-gnu/SHA256SUMS.part a091e4dce267164a757067c279d7df609b112867944a3689fbd9b6fefa9dcd40 guix-build-8d25926643cb/output/powerpc64-linux-gnu/bitcoin-8d25926643cb-powerpc64-linux-gnu-debug.tar.gz 195c8d6596acfe78f1f250b4ec8598c9b8c1e3420b96b8dd16529f194f4f5602 guix-build-8d25926643cb/output/powerpc64-linux-gnu/bitcoin-8d25926643cb-powerpc64-linux-gnu.tar.gz 78fe3b028498818532ce2120d7262ddddd2533ee58d3d04fc7908d2a3384eac9 guix-build-8d25926643cb/output/powerpc64le-linux-gnu/SHA256SUMS.part 52fba781d6637751500febb1470dd653933da2cf8f7ae052416438a80a925d7a guix-build-8d25926643cb/output/powerpc64le-linux-gnu/bitcoin-8d25926643cb-powerpc64le-linux-gnu-debug.tar.gz 4cb12dce174a58ccb46381efcfc27f9d49ea275cfbda68f334a73c2fa92fe18e guix-build-8d25926643cb/output/powerpc64le-linux-gnu/bitcoin-8d25926643cb-powerpc64le-linux-gnu.tar.gz c8f5eb94bbf9577f59ae2a679e6807ba65380356656cc335d1e13390e17878bc guix-build-8d25926643cb/output/riscv64-linux-gnu/SHA256SUMS.part 97e28f9033341f08ab16a4116d593fb4b63da8838da62af49f64f9d6f26b8276 guix-build-8d25926643cb/output/riscv64-linux-gnu/bitcoin-8d25926643cb-riscv64-linux-gnu-debug.tar.gz 1fb6c289347fbc6c13a7eb189b2a1bdf84c94b16f38dd786bdcc2a8ee0b6d136 guix-build-8d25926643cb/output/riscv64-linux-gnu/bitcoin-8d25926643cb-riscv64-linux-gnu.tar.gz 1080f0e8e6ee9aff33b47fe90f3f55df368cd3b464bc60eebc1e0092a37bc957 guix-build-8d25926643cb/output/x86_64-apple-darwin/SHA256SUMS.part 7cd7d4b68e2ec47bcec035b77c956a832453ac1589c7f552285623f5ea84a354 guix-build-8d25926643cb/output/x86_64-apple-darwin/bitcoin-8d25926643cb-x86_64-apple-darwin-unsigned.dmg 9283cfe406694d7f1edc78acaddd817aebb7f2c7397b21687e4410ab71078773 guix-build-8d25926643cb/output/x86_64-apple-darwin/bitcoin-8d25926643cb-x86_64-apple-darwin-unsigned.tar.gz e96b0ad265db5f990c087bc857272c7d6ede32cd89dc8fd201c4fd25b9bdf072 guix-build-8d25926643cb/output/x86_64-apple-darwin/bitcoin-8d25926643cb-x86_64-apple-darwin.tar.gz a3ff233ea536e9e6c19eecca54a7703dad901b4aab9b9606a2203af844be7f57 guix-build-8d25926643cb/output/x86_64-linux-gnu/SHA256SUMS.part b48d92a330f20cc9cfb8d9574c2e0190e5ee497120f7dbd0bb00e5001605ef3a guix-build-8d25926643cb/output/x86_64-linux-gnu/bitcoin-8d25926643cb-x86_64-linux-gnu-debug.tar.gz d06d54c2bdd66e7031e1ed2da411343c665763e43312eee87a7f575db799a450 guix-build-8d25926643cb/output/x86_64-linux-gnu/bitcoin-8d25926643cb-x86_64-linux-gnu.tar.gz 7164b2b3caa3ca56dc10098474325f92c328355fee7f1464069b99b72853bab1 guix-build-8d25926643cb/output/x86_64-w64-mingw32/SHA256SUMS.part f01a344bd8715479955911a465097bb196daffa04c8c334cab41e22529ad2f5c guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64-debug.zip 5ae8999fdc66ef75ce87e1824b3fcbbb2278104f3e2a7b50a90d10e92c974673 guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64-setup-unsigned.exe 6beb94dbf0c843d80480901880074e7d1bdf9b9dce2153ac8980c27b484d1fe5 guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64-unsigned.tar.gz de36858355e4053cd2870782072a073d1d2e39627b4ff7c3d48699f419966d20 guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64.zip ``` Guix Build (arm64): ```bash 51703356187e978d0b6a2cafd7533554b9860f1bf934547ea2fa6e673d8c2009 guix-build-8d25926643cb/output/arm-linux-gnueabihf/SHA256SUMS.part e66aad4e90f7c88e839fe9c8c322b204d9c480812e5890d6b385aeb67893e3a0 guix-build-8d25926643cb/output/arm-linux-gnueabihf/bitcoin-8d25926643cb-arm-linux-gnueabihf-debug.tar.gz 382c99aa79a92d9e6854490ee184ac8e7bf1ac1b9ddcab98421d959ba4c4797c guix-build-8d25926643cb/output/arm-linux-gnueabihf/bitcoin-8d25926643cb-arm-linux-gnueabihf.tar.gz 82f81c109a11a16cf56a5d4029cf7636c34a351f39902ec99d9b2da40e5b3073 guix-build-8d25926643cb/output/arm64-apple-darwin/SHA256SUMS.part e62438b4cb68a5123f2e91c8711b5cd24e1567693b142174321a79c5c9c3e040 guix-build-8d25926643cb/output/arm64-apple-darwin/bitcoin-8d25926643cb-arm64-apple-darwin-unsigned.dmg 3bd0bc61c2f1d8b30db8dc0fb25a6f73d95416115ccd101922e962852d2f2fd4 guix-build-8d25926643cb/output/arm64-apple-darwin/bitcoin-8d25926643cb-arm64-apple-darwin-unsigned.tar.gz 3b987efd79203b0257b1590926106b890d08a0e3d3b0c9ea6e3194c3f4d9cab5 guix-build-8d25926643cb/output/arm64-apple-darwin/bitcoin-8d25926643cb-arm64-apple-darwin.tar.gz 71d86ffaf1b54b1dfb0ee6a978c2d3c800742d82a7845967becdd88b1ba4cbf4 guix-build-8d25926643cb/output/dist-archive/bitcoin-8d25926643cb.tar.gz bcfbdf2389275b7982a0f0f02ce6a0032b6c4688a56dee882f0b4da5985bffa5 guix-build-8d25926643cb/output/powerpc64-linux-gnu/SHA256SUMS.part 61ec34aa20de6281c4749961e09cbdd133e46397bee882fd2c211554b311bc9b guix-build-8d25926643cb/output/powerpc64-linux-gnu/bitcoin-8d25926643cb-powerpc64-linux-gnu-debug.tar.gz 8b96152ec8dfde4f2b2046f2c54bd13ea066170a4eaa6b63054fe6e9eabf0427 guix-build-8d25926643cb/output/powerpc64-linux-gnu/bitcoin-8d25926643cb-powerpc64-linux-gnu.tar.gz 78ae0da590b2c174e3875aceccbd8b3195578408770f4c1c29c2d135aa19d638 guix-build-8d25926643cb/output/powerpc64le-linux-gnu/SHA256SUMS.part e755b3cf2a3c2a1671d3d312ab8b45abc0e544e978b01c7645939bb2d5d16a63 guix-build-8d25926643cb/output/powerpc64le-linux-gnu/bitcoin-8d25926643cb-powerpc64le-linux-gnu-debug.tar.gz 4bdafdc83c7a81cfa152036a5366c09862559002257e1727ff7f930e7f6e6c91 guix-build-8d25926643cb/output/powerpc64le-linux-gnu/bitcoin-8d25926643cb-powerpc64le-linux-gnu.tar.gz 64bfcba320ebcc81bdaecf932807a6f487b32a52b7ddccb6c794c4f2e0c3e1c3 guix-build-8d25926643cb/output/riscv64-linux-gnu/SHA256SUMS.part c0eebeb9e1fb441db397c33b833138facdde0e984c5a3d48b3957487867c98c7 guix-build-8d25926643cb/output/riscv64-linux-gnu/bitcoin-8d25926643cb-riscv64-linux-gnu-debug.tar.gz 97b2b432cf7c628c9960768057f9c46754c01860c3e77a8baca29465f18ad120 guix-build-8d25926643cb/output/riscv64-linux-gnu/bitcoin-8d25926643cb-riscv64-linux-gnu.tar.gz 1080f0e8e6ee9aff33b47fe90f3f55df368cd3b464bc60eebc1e0092a37bc957 guix-build-8d25926643cb/output/x86_64-apple-darwin/SHA256SUMS.part 7cd7d4b68e2ec47bcec035b77c956a832453ac1589c7f552285623f5ea84a354 guix-build-8d25926643cb/output/x86_64-apple-darwin/bitcoin-8d25926643cb-x86_64-apple-darwin-unsigned.dmg 9283cfe406694d7f1edc78acaddd817aebb7f2c7397b21687e4410ab71078773 guix-build-8d25926643cb/output/x86_64-apple-darwin/bitcoin-8d25926643cb-x86_64-apple-darwin-unsigned.tar.gz e96b0ad265db5f990c087bc857272c7d6ede32cd89dc8fd201c4fd25b9bdf072 guix-build-8d25926643cb/output/x86_64-apple-darwin/bitcoin-8d25926643cb-x86_64-apple-darwin.tar.gz 74ee4531452feffa90874cfd8c96d3b941627bd5cbb4a6977d393a2b08a13e47 guix-build-8d25926643cb/output/x86_64-linux-gnu/SHA256SUMS.part 15815a3269a2a71047afe635765f79d9b31ba08b62cad09bad17ca865f522afb guix-build-8d25926643cb/output/x86_64-linux-gnu/bitcoin-8d25926643cb-x86_64-linux-gnu-debug.tar.gz 6146bf0ffba945d4a21e36fa0857c5bedc091e953a79ad5782008d759b579c94 guix-build-8d25926643cb/output/x86_64-linux-gnu/bitcoin-8d25926643cb-x86_64-linux-gnu.tar.gz 4c0fbf62b7b76ec14b41eb10cd2f0ae4ef27028044e029a02abad3aa290a6123 guix-build-8d25926643cb/output/x86_64-w64-mingw32/SHA256SUMS.part 241955fd1c457ba4206814bfdd7652cd4975d68ec905ec9d80a45d3ddd69e0bf guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64-debug.zip 5ae8999fdc66ef75ce87e1824b3fcbbb2278104f3e2a7b50a90d10e92c974673 guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64-setup-unsigned.exe 6beb94dbf0c843d80480901880074e7d1bdf9b9dce2153ac8980c27b484d1fe5 guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64-unsigned.tar.gz 0eeacc664ad0fb89f177fa6271f8aa573e177804121a7ef2391f06cd56cdc74a guix-build-8d25926643cb/output/x86_64-w64-mingw32/bitcoin-8d25926643cb-win64.zip ``` ACKs for top commit: laanwj: Code review ACK 8d25926 Tree-SHA512: 2da8037cc60b28ff210c392785e750f33893d9e0b223ac8c5a66cc3eec8f0558c2af7c31494b6075974180fd0fcf34675b5071fe19ad112f379a7489ef1a1a9a
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 10, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 10, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 13, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 17, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 19, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 28, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 30, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
May 31, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
Jun 3, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
Jun 4, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
Jun 4, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
Jun 6, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
Jun 7, 2023
kwvg
added a commit
to kwvg/dash
that referenced
this pull request
Jun 7, 2023
PastaPastaPasta
added a commit
to dashpay/dash
that referenced
this pull request
Jun 8, 2023
backport: merge bitcoin#18434, bitcoin#21664, bitcoin#20608, bitcoin#21871, bitcoin#21255, bitcoin#21428, bitcoin#21889, bitcoin#22070, bitcoin#22392, bitcoin#23535, bitcoin#23811, bitcoin#23838, bitcoin#23839, bitcoin#23148, bitcoin#20422, bitcoin#24561, bitcoin#26075, bitcoin#25458, partial bitcoin#22381 (guix backports: part 1)
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Even though it's (re)built infrequently, the LIEF build currently runs with a single job, and thus takes much longer to build than it should. This change modifies our packge so that the cmake invocation is done with multiple jobs, using
parallel-job-count. On a 6 core arm64 machine, this results in a ~80% speed up, and reduces the time taken to build the LIEF packge from ~19m to ~4m.Guix Build (x86_64):
Guix Build (arm64):