Backports 0.19 guix #4634
Closed
Backports 0.19 guix #4634
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Munkybooty
force-pushed
the
backports-0.19-guix
branch
from
954ffe3 to
bb25f1e
Compare
UdjinM6
requested changes
Dec 22, 2021
Munkybooty
force-pushed
the
backports-0.19-guix
branch
from
bb25f1e to
98e18ab
Compare
UdjinM6
previously approved these changes
Dec 25, 2021
UdjinM6
left a comment
UdjinM6
left a comment
There was a problem hiding this comment.
utACK assuming we are going to dashify this later
Member
|
@Munkybooty do future (non-guix) backports rely on this? Or can we delay this until v19? |
Author
|
I think we should be safe delaying this, all the files are basically isolated and guix seems to be untouched until v20 backports. The only merge conflicts I can see appearing are with qt.mk other than that, it should be fine |
UdjinM6
requested changes
Feb 16, 2022
Munkybooty
force-pushed
the
backports-0.19-guix
branch
from
7383263 to
6698ca4
Compare
|
This pull request has conflicts, please rebase. |
Munkybooty
force-pushed
the
backports-0.19-guix
branch
from
6698ca4 to
5651811
Compare
751549b contrib: guix: Additional clarifications re: substitutes (Carl Dong) cd3e947 contrib: guix: Various improvements. (Carl Dong) 8dff3e4 contrib: guix: Clarify SOURCE_DATE_EPOCH. (Carl Dong) 3e80ec3 contrib: Add deterministic Guix builds. (Carl Dong) Pull request description: ~~**This post is kept updated as this project progresses. Use this [latest update link](bitcoin#15277 (comment)) to see what's new.**~~ Please read the `README.md`. ----- ### Guix Introduction This PR enables building bitcoin in Guix containers. [Guix](https://www.gnu.org/software/guix/manual/en/html_node/Features.html) is a transactional package manager much like Nix, but unlike Nix, it has more of a focus on [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) and [reproducibility](https://www.gnu.org/software/guix/blog/tags/reproducible-builds/) which are attractive for security-sensitive projects like bitcoin. ### Guix Build Walkthrough Please read the `README.md`. [Old instructions no. 4](bitcoin#15277 (comment)) [Old instructions no. 3](bitcoin#15277 (comment)) [Old instructions no. 2](bitcoin#15277 (comment)) <details> <summary>Old instructions no. 1</summary> In this PR, we define a Guix [manifest](https://www.gnu.org/software/guix/manual/en/html_node/Invoking-guix-package.html#profile_002dmanifest) in `contrib/guix/manifest.scm`, which declares what packages we want in our environment. We can then invoke ``` guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` To have Guix: 1. Build an environment containing the packages we defined in our `contrib/guix/manifest.scm` manifest from the Guix bootstrap binaries (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html) for more details). 2. Start a container with that environment that has no network access, and no access to the host's filesystem except to the `pwd` that it was started in. 3. Drop you into a shell in that container. > Note: if you don't want to wait hours for Guix to build the entire world from scratch, you can eliminate the `--no-substitutes` option to have Guix download from available binary sources. Note that this convenience doesn't necessarily compromise your security, as you can check that a package was built correctly after the fact using `guix build --check <packagename>` Therefore, we can perform a build of bitcoin much like in Gitian by invoking the following: ``` make -C depends -j"$(nproc)" download && \ cat contrib/guix/build.sh | guix environment --manifest=contrib/guix/manifest.scm --container --pure --no-grafts --no-substitutes ``` We don't include `make -C depends -j"$(nproc)" download` inside `contrib/guix/build.sh` because `contrib/guix/build.sh` is run inside the container, which has no network access (which is a good thing). </details> ### Rationale I believe that this represents a substantial improvement for the "supply chain security" of bitcoin because: 1. We no longer have to rely on Ubuntu for our build environment for our releases ([oh the horror](https://github.com/bitcoin/bitcoin/blob/72bd4ab867e3be0d8410403d9641c08288d343e3/contrib/gitian-descriptors/gitian-linux.yml#L10)), because Guix builds everything about the container, we can perform this on almost any Linux distro/system. 2. It is now much easier to determine what trusted binaries are in our supply chain, and even make a nice visualization! (see [bootstrappability](https://www.gnu.org/software/guix/manual/en/html_node/Bootstrapping.html)). 3. There is active effort among Guix folks to minimize the number of trusted binaries even further. OriansJ's [stage0](https://github.com/oriansj/stage0), and janneke's [Mes](https://www.gnu.org/software/mes/) all aim to achieve [reduced binary boostrap](http://joyofsource.com/reduced-binary-seed-bootstrap.html) for Guix. In fact, I believe if OriansJ gets his way, we will end up some day with only a single trusted binary: hex0 (a ~500 byte self-hosting hex assembler). ### Steps to Completion - [x] Successfully build bitcoin inside the Guix environment - [x] Make `check-symbols` pass - [x] Do the above but without nasty hacks - [x] Solve some of the more innocuous hacks - [ ] Make it cross-compile (HELP WANTED HERE) - [x] Linux - [x] x86_64-linux-gnu - [x] i686-linux-gnu - [x] aarch64-linux-gnu - [x] arm-linux-gnueabihf - [x] riscv64-linux-gnu - [ ] OS X - [ ] x86_64-apple-darwin14 - [ ] Windows - [ ] x86_64-w64-mingw32 - [ ] Maybe make importer for depends syntax - [ ] Document build process for future releases - [ ] Extra: Pin the revision of Guix that we build with with Guix [inferiors](https://www.gnu.org/software/guix/manual/en/html_node/Inferiors.html) ### Help Wanted [Old content no. 3](bitcoin#15277 (comment)) [Old content no. 2](bitcoin#15277 (comment)) <details> <summary>Old content no. 1</summary> As of now, the command described above to perform a build of bitcoin a lot like Gitian works, but fails at the `check-symbols` stage. This is because a few dynamic libraries are linked in that shouldn't be. Here's what `ldd src/bitcoind` looks like when built in a Guix container: ``` linux-vdso.so.1 (0x00007ffcc2d90000) libdl.so.2 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libdl.so.2 (0x00007fb7eda09000) librt.so.1 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/librt.so.1 (0x00007fb7ed9ff000) libstdc++.so.6 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libstdc++.so.6 (0x00007fb7ed87c000) libpthread.so.0 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libpthread.so.0 (0x00007fb7ed85b000) libm.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libm.so.6 (0x00007fb7ed6da000) libgcc_s.so.1 => /gnu/store/4sqps8dczv3g7rwbdibfz6rf5jlk7w90-gcc-5.5.0-lib/lib/libgcc_s.so.1 (0x00007fb7ed6bf000) libc.so.6 => /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/libc.so.6 (0x00007fb7ed506000) /gnu/store/h90vnqw0nwd0hhm1l5dgxsdrigddfmq4-glibc-2.28/lib/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007fb7ee3a0000) ``` And here's what it looks in one of our releases: ``` linux-vdso.so.1 (0x00007ffff52cd000) libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f87726b4000) librt.so.1 => /usr/lib/librt.so.1 (0x00007f87726aa000) libm.so.6 => /usr/lib/libm.so.6 (0x00007f8772525000) libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007f877250b000) libc.so.6 => /usr/lib/libc.so.6 (0x00007f8772347000) /lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f8773392000) ``` ~~I suspect it is because my script does not apply the gitian-input patches [described in the release process](https://github.com/bitcoin/bitcoin/blob/master/doc/release-process.md#fetch-and-create-inputs-first-time-or-when-dependency-versions-change) but there is no description as to how these patches are applied.~~ It might also be something else entirely. Edit: It is something else. It appears that the gitian inputs are only used by [`gitian-win-signer.yml`](https://github.com/bitcoin/bitcoin/blob/d6e700e40f861ddd6743f4d13f0d6f6bc19093c2/contrib/gitian-descriptors/gitian-win-signer.yml#L14) </details> ### How to Help 1. Install Guix on your distro either [from source](https://www.gnu.org/software/guix/manual/en/html_node/Requirements.html) or perform a [binary installation](https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html#Binary-Installation) 2. Try out my branch and the command described above! ACKs for top commit: MarcoFalke: Thanks for the replies. ACK 751549b laanwj: ACK 751549b Tree-SHA512: 50e6ab58c6bda9a67125b6271daf7eff0ca57d0efa8941ed3cd951e5bf78b31552fc5e537b1e1bcf2d3cc918c63adf19d685aa117a0f851024dc67e697890a8d
0065ead contrib: guix: Remove ssp spec file hack (Carl Dong) 0093a58 contrib: guix: More robust search paths, add checks (Carl Dong) Pull request description: See commit messages for more details ACKs for top commit: fanquake: ACK 0065ead Tree-SHA512: fde04005fb31cd4b75b80da4936a7c394f63f0b3bdcc33c20c99e05604a63efd9c850a8d097030ff0bf4b4e83f1a9997fc4621ce291ebcecd8397893447600a7
Munkybooty
force-pushed
the
backports-0.19-guix
branch
from
5651811 to
700ad49
Compare
|
This pull request has conflicts, please rebase. |
Collaborator
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.