-
Notifications
You must be signed in to change notification settings - Fork 686
Build and packaging strategy 2022 05 31
Erik Moeller edited this page Jun 1, 2022
·
1 revision
- Date: 2022-05-31
- Attendees: Kunal, Kev, Kushal, Allie, Erik, Ro, Alex, Giulio, Tina, Gonzalo, Cory
- Slides: https://docs.google.com/presentation/d/1Ntpeo_qlchY9ECwGcs5x6Cqzb-IijjSbn-WsI3B4AvU/edit?usp=sharing
To discuss:
- architecture of our packaging-related repositories
- potential improvements to the build workflow
- supply chain attack mitigation for dependencies
- standardization of the build workflow across SD Server/ SD Workstation
"Packing and release improvements"
- NB. "If you've ever wondered by debhelper runs on Perl...the entire world runs on Perl."
- We primarily build via dh-virtualenv, to set up the virtualenv we build within.
- This is the canonical Debian toolchain, though there are higher-level tools that build on top of them.
- "debian/rules" is dh-* specific Make overrides.
- The usual way in Debian to avoid host-system clutter is to build in a chroot.
- Con: We do a lot of stuff slowly by hand (s/hand/Docker), non-reproducibly.
- Pro: "make build-debs" is one entrypoint to build everything.
Among other things:
- Uses localwheels, for verifiability/reproducibility based on our diff reviews.
-
Kushal: Python will try to move to reproducible wheels upstream too.
-
Cory: Differences between chroot/Docker/dispVM containerization?
-
Kunal: Not really a security concern; more about cleanliness of build. E.g., a dispVM is going to be much heavier than either a chroot or a Docker image, because it has to have all the Qubes dependencies baked in.
-
Alex: Could always chroot in Docker or dispVMs to run Debian's chroot workflows and tools under Docker/Xen's better isolation.
-
- "Networkless" means "no downloads after apt {update,install}".
- After question from Ro: Multiple distribution versions should be supportable from a single branch, probably even the same "debian/" directory, just built in different chroots.
-
Kushal: Networkless builds have been a long-time dream.
-
Cory: I like how much is automated here or reduced to verification steps. I'm wondering if there's anything in our current QA and release procedures that this requires us to revisit assumptions around?
- Kunal: apt-test → apt-prod workflows/conventions/expectations. (apt-test like Git "main", expected to be publishable). Would just need to be define carefully during the release cycles these changes are made across.
-
Ro: What Workstation components will be affected?
- Kunal: One "debian/" directory can spit out multiple of those components.
[5 min break]
Core issues/themes to resolve: - Build env standardization (chroot vs. Docker vs. dispVM) - Role of CI - Future of localwheels - Future of apt-qa - which packages would we group together vs make separately releasable?
- Alex: chroots are not quite as isolated: there have been symlink- and process-level injections, for example. Shouldn't be possible in Docker. chroots weren't designed for security isolation.
- Kunal: We trust everything that runs in the packaging process. The goal of the chroot is to prevent the host environment from leaking in (plus a little bit of cross-distro stuff).
- Erik: Hypothesis: The greatest security guarantee comes from reproducibility?
- Kunal: Yes. Currently we trust developer builds; in this case we'd trust CI builds verified against developer builds.
- Kunal: Always use a Debian stable chroot.
- Erik: Are folks comfortable with the chroot?
- Kev: Not without more research versus Docker.
- Erik: Why isn't securedrop-app-code reproducible?
- Kunal: Download stuff directly from PyPI; skip some debhelper steps; hard to debug at a clean breakpoint in Molecule.
- Erik: Anything you like in the Ansible-based toolchain?
- Kunal: Single entrypoint. But we could do that in standard tooling: one source package→ multiple binary packages.
- Erik: Incremental step: make the Server's build process more like the Workstation's. Does that also require localwheels?
- Kunal: Not required, but we should.
- Erik: Summary: We don't build "manylinux" wheels, so we can't verify them against PyPI binary wheels.
- Kushal: Most large companies build and maintain their own wheels .
- Erik: So, for bit-for-bit verification, we'd have to set up our own manylinux build infrastructure?
- Kunal: Yep, e.g. cibuildwheel. I think we should keep doing localwheels until the Python world better figures out manylinux reproducibility.
- Allie: What about the "none-any" universal wheel alongside manylinux wheel
- Kushal: The biggest problem with manylinux wheels is that they have to run from old CentOS (6?) to newer systems. manylinux wheels don't support reproducibility, because they're going to package random libraries from those old systems. ("none-any" universal wheels are pure Python, no extensions.)
- Kev: What would the flow of packages be from dev to production?
- Kunal: RCs get published to apt-test. Final release artifacts also published to apt-test. Then copy that directory from apt-test to apt-prod and sign the artifacts with the release key. Makes it possible to diff between apt-test and apt-prod.
- Kev: Would we lose the ability to prevent artifacts from apt-test winding up on apt-prod?
- Kunal: No, you could copy over only the artifacts you wanted. The point is that nothing should come to apt-prod without first coming to apt-prod.
- Erik: So then the only value of apt-qa is testing the prod signing procedure. Is that worth the value?
- Kev: No, it would be a catastrophic failure in any case.
- Allie: Worried about syncig apt with apt-test, what if (1) RC4 is good to be released, so we remove all the RCs from the package, then we find a bug and want to revert to RC2.
- Kunal: Check out the old RC2 tag and build a new package. We have to use [monotonic] version numbers.
Kunal will roadmap an epic....
- Kunal: The Workstation is already "halfway to the dream", so I'd start with the server.
- Erik: Other priorities?
- Allie: Cleanup to support both buster and bullseye (for Qubes 4.1).