Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

enhancements: Propose CoreOS layering #7

Merged
merged 5 commits into from
Nov 22, 2021

Conversation

cgwalters
Copy link
Member

@cgwalters cgwalters commented Oct 19, 2021

This enhancement proposes:

  • A fundamental new addition to ostree/rpm-ostree, which is support for directly pulling and updating the OS from container images (while keeping all existing functionality, per-node package layering and our existing support for pulling via ostree on the wire).
  • Documentation for generating derived (layered) images from the pristine CoreOS base image.
  • Support for switching a FCOS system to use a custom image on firstboot via Ignition
  • zincati will continue to perform upgrades by inspecting the upgrade graph from the base image.

Work tracker

Trying this out

You'll need rpm-ostree v2021.14 or newer.

Current snapshot container base image: quay.io/cgwalters/fcos


New rpm-ostree architecture diagram

            ┌─────────────────┐
            │                 │
         ┌──┤   rpm-ostree    ├────┐
         │  │                 │    │
         │  └─────────────────┘    │
         │                         │
         │                         │
┌────────▼────────┐       ┌────────▼────────┐
│                 │       │                 │
│     libdnf      │       │  ostree-rs-ext  ├────────────┐
│                 │       │                 │            │
└────────┬────────┘       └────────┬────────┘            │
         │                         │                     │
┌────────▼────────┐       ┌────────▼────────┐   ┌────────▼─────────┐
│                 │       │                 │   │                  │
│     librpm      │       │    ostree-rs    │   │  imageproxy-rs   │
│                 │       │                 │   │                  │
└─────────────────┘       └────────┬────────┘   └───────────┬──────┘
                                   │                        │
                          ┌────────▼────────┐   ┌───────────▼──────────────────┐
                          │                 │   │ (forked process)             │
                          │     ostree      │   │                              │
                          │                 │   │     ┌──────────────────┐     │
                          └─────────────────┘   │     │                  │     │
                                                │     │     skopeo       │     │
                                                │     │                  │     │
                                                │     └────────┬─────────┘     │
                                                │              │               │
                                                │     ┌────────▼─────────┐     │
                                                │     │                  │     │
                                                │     │ containers/image │     │
                                                │     │                  │     │
                                                │     └──────────────────┘     │
                                                │                              │
                                                │                              │
                                                │                              │
                                                └──────────────────────────────┘

Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this is an interesting idea!

I wonder if we should have an intermediate enhancement before this one which is about officially shipping FCOS as a container and supporting rebasing to it and updating. And then from there we could have this enhancement focus more precisely on the idea of derivation.

os/coreos-layering.md Outdated Show resolved Hide resolved
os/coreos-layering.md Outdated Show resolved Hide resolved
@jlebon
Copy link
Member

jlebon commented Oct 20, 2021

Worth discussing also the intersection between derivation and Ignition. There's clearly a lot of overlap there in terms of configuration.

@cgwalters
Copy link
Member Author

I wonder if we should have an intermediate enhancement before this one which is about officially shipping FCOS as a container and supporting rebasing to it and updating.

Not opposed, but in the end "encapsulation" wasn't intended to be a user-visible thing until perhaps the very end, and it's not that interesting. This is much more consequential and acts as a driver for it, and while they're logically separate from the user PoV I think we might as well do them together.

@LorbusChris
Copy link

fyi @vrutkovs, I think this'll be great for building okd-machine-os

@cgwalters
Copy link
Member Author

cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 2, 2021
cgwalters added a commit to cgwalters/coreos-assembler that referenced this pull request Nov 2, 2021
To pick up new ostree 2021.5 for coreos/enhancements#7
jlebon pushed a commit to coreos/coreos-assembler that referenced this pull request Nov 2, 2021
To pick up new ostree 2021.5 for coreos/enhancements#7
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 2, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 4, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 4, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 5, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 5, 2021
cgwalters added a commit to cgwalters/fedora-coreos-config that referenced this pull request Nov 5, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 5, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 6, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 6, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 8, 2021
cgwalters added a commit to cgwalters/fedora-coreos-config that referenced this pull request Nov 8, 2021
cgwalters added a commit to cgwalters/fedora-coreos-config that referenced this pull request Nov 8, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 8, 2021
cgwalters added a commit to cgwalters/rpm-ostree that referenced this pull request Nov 8, 2021
@vrutkovs
Copy link
Member

+1 from me, we're certainly interested in this in OKD

os/coreos-layering.md Outdated Show resolved Hide resolved
os/coreos-layering.md Outdated Show resolved Hide resolved
os/coreos-layering.md Outdated Show resolved Hide resolved

# Butane as a declarative input format for layering

We must support `Dockerfile`, because it's the lowest common denominator for the container ecosystem, and is accepted as input for many tools.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be nice if there was a halfway approach where we do support the Dockerfile approach but the only supported operation is still to just apply a Butane config, potentially with some locally referenced files that were copied from a previous build stage.

Personally, my primary concerns with the "freehand" Dockerfile approach is that:

  1. It's too easy to make drastic changes. E.g. in rpm-ostree, we have this conscious decision to separate out pure package layering and base package overrides. This distinction does not exist here, and removing these guardrails means it becomes much easier for users to shoot themselves in the foot. I guess one thing we could do is have rpm-ostree detect when content was modified instead of purely added and require a --allow-base-overrides or something?
  2. It makes the configuration story messier. With this, you can now configure at both image build time and Ignition boot time. And they each use completely different languages. So I think there's potential there to cause user confusion. It also impacts support, which now needs to understand more types of inputs.

Focusing on Butane configs as the layering mechanism in my opinion helps with both of those because it's more declarative and so easier to analyze, and it's the same configuration language we've been speaking so far.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be clear, I'm OK with this too, and I know we can always tweak things going forward (though it's always harder to add restrictions than it is to remove them). Just wanted to point out what IMO are some non-negligible risks of this approach.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have this conscious decision to separate out pure package layering and base package overrides. This distinction does not exist here,

That's not quite true, because we will still have the original base image. Reverting back to the "golden FCOS base" will also be a just an rpm-ostree rebase away.

Additionally, because we will have yum|dnf -> rpm-ostree in this image (xref https://coreos.github.io/rpm-ostree/cliwrap/) we can actually still impose the install vs override semantic.

Further, tooling can perform filesystem and rpmdb diffing between layers - we have all that code today.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And they each use completely different languages.

Also, today one can use any tool to generate Ignition, which definitely happens (e.g. openshift-install generates Ignition via some custom Terraform stuff) and we support that.

More broadly I think we are in a position of needing to support "low level" as well as arbitrary configuration mechanisms, but we also try to have opinions on higher level tooling.

So here Dockerfile is that lowest common denominator for arbitrary, low level (kinda) mechanism - but we can streamline higher level workflows inside of that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have this conscious decision to separate out pure package layering and base package overrides. This distinction does not exist here,

That's not quite true, because we will still have the original base image. Reverting back to the "golden FCOS base" will also be a just an rpm-ostree rebase away.

Right, but the power a user wields in their Dockerfile sees it all as just one filesystem. E.g. nothing stops them from doing COPY my-libc.so /usr/lib64/libc.so and the build system will be happy to build that. Whereas if it's Butane based, we can immediately tell they're modifying base content and fail the build. So the failure happens at image build time instead of deploy and reboot time. Of course, we do want to provide flexibility to modify base content, but IMO it should be explicitly opted in.

Combining this with ostreedev/ostree-rs-ext#159, maybe we could have this ostree container finalize step take that switch? E.g. ostree container finalize --allow-overrides.

Additionally, because we will have yum|dnf -> rpm-ostree in this image (xref coreos.github.io/rpm-ostree/cliwrap) we can actually still impose the install vs override semantic.

Hmm, can you expand on the UX you're thinking of here? dnf install will automatically upgrade an already installed package so we'd have to require some added switch (related: coreos/rpm-ostree#2844 (comment)).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, can you expand on the UX you're thinking of here? dnf install will automatically upgrade an already installed package so we'd have to require some added switch (related: coreos/rpm-ostree#2844 (comment)).

Well, the cliwrap status quo today errors actually when someone types dnf install. I know this is a perpetually confusing thing but so far cliwrap does not involve necessarily invoking traditional yum|dnf logic. It opens the door to that of course.

So specifically if e.g. someone times RUN yum install usbguard in their Dockerfile, we would error out, and they'd have to do RUN rpm-ostree install usbguard which would run through all of the same logic that exists today which would only layer, not upgrade dependencies. (There's...a lot implied in this whole "rpm-ostree in container builds" flow, really worth splitting out to a separate issue; will look at that)

Copy link
Member Author

@cgwalters cgwalters Nov 17, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The real status quo right now to be clear is that dnf install usbguard will give "command not found" and rpm-ostree install usbguard will error out because you're in a container. And rpm -Uvh https://example.com/usbguard.rpm will work.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

> podman run --rm -ti quay.io/cgwalters/fcos bash
bash-5.1# dnf install usbguard
bash: dnf: command not found
bash-5.1# rpm-ostree install usbguard
error: This system was not booted via libostree; found $container=podman environment variable.
rpm-ostree is designed to manage host systems, not containers.

bash-5.1# 

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So specifically if e.g. someone times RUN yum install usbguard in their Dockerfile, we would error out, and they'd have to do RUN rpm-ostree install usbguard which would run through all of the same logic that exists today which would only layer, not upgrade dependencies.

Ack, I like that.

What I'm trying to make sure here is that OCP customers who will be interacting with this feature are still subject to basic rules by default so that our QA efforts aren't trivially invalidated. I.e. I don't want them to be able to change /usr base content just as easily as they can /etc content. If the only way OCP customers interact with this is through Butane or MCs for example, then I think that's fine. Otherwise, I think we should make them opt into modifying base content explicitly (and maybe e.g. we only support them doing this if they have a support exception).

Taking this feature in isolation (e.g. for FCOS users), typing rpm-ostree rebase $my_pullspec already implies some risk acceptance (though there's still value in keeping guardrails there too of course). But in the OCP case, the MCO will be what's running that command.

Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway, don't want to block on this necessarily, but I think we should keep in mind this idea of distinguishing between modifying vs adding when designing and implementing this and higher levels. Ideally we'd mention it somewhere in the proposal at least.

@cgwalters
Copy link
Member Author

@cgwalters
Copy link
Member Author

Do you have any suggested text on this? Pasting as a comment works, or a diff. I added some text in the Dockerfile section.

There's really a whole lot that could be written here around debuggability/introspection. One thing that ostree can provide much faster than the default containers/storage stack is filesystem level diffs, because it's all content addressed.

Hmm. I guess in theory too we could actually require something like
LABEL ostree.override=true to support overwriting content in previous layers?
That could be somewhat easily enforced in the ostree-container stack.

@cgwalters
Copy link
Member Author

Anyway, don't want to block on this necessarily, but I think we should keep in mind this idea of distinguishing between modifying vs adding when designing and implementing this and higher levels. Ideally we'd mention it somewhere in the proposal at least.

One thing I want to say is this is a super useful discussion! I mean, I have opinions on a lot of this stuff, but for sure there is a vast space here filled with nuance and important details.

I think this enhancement is just the start to a story, not an end.

os/coreos-layering.md Outdated Show resolved Hide resolved
os/coreos-layering.md Show resolved Hide resolved
cgwalters and others added 2 commits November 22, 2021 10:17
Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
Co-authored-by: Jonathan Lebon <jonathan@jlebon.com>
@cgwalters cgwalters merged commit 9cdcc68 into coreos:main Nov 22, 2021
cgwalters added a commit to cgwalters/os that referenced this pull request Nov 30, 2021
cgwalters added a commit to cgwalters/os that referenced this pull request Nov 30, 2021
cgwalters added a commit to cgwalters/os that referenced this pull request Nov 30, 2021
cgwalters added a commit to cgwalters/os that referenced this pull request Jan 17, 2022
cgwalters added a commit to cgwalters/os that referenced this pull request Feb 1, 2022
cgwalters added a commit to cgwalters/os that referenced this pull request Feb 1, 2022
cgwalters added a commit to cgwalters/enhancements that referenced this pull request Feb 10, 2022
**NOTE: Nothing in this proposal should be viewed as final.
It is highly likely that details will change.  It is quite possible
that larger architectural changes will be made as well.**

Change RHEL CoreOS as shipped in OpenShift to be a "base image" that can
be used as in layered container builds and then booted.  This will allow
custom 3rd party agents delivered via RPMs installed in a container
build.  The MCO will roll out and monitor these custom builds the same
way it does for the "pristine" CoreOS image today.

This is the OpenShift integration of [ostree native containers](https://fedoraproject.org/wiki/Changes/OstreeNativeContainer) or [CoreOS layering](coreos/enhancements#7) via the MCO.
cgwalters added a commit to cgwalters/enhancements that referenced this pull request Feb 10, 2022
**NOTE: Nothing in this proposal should be viewed as final.
It is highly likely that details will change.  It is quite possible
that larger architectural changes will be made as well.**

Change RHEL CoreOS as shipped in OpenShift to be a "base image" that can
be used as in layered container builds and then booted.  This will allow
custom 3rd party agents delivered via RPMs installed in a container
build.  The MCO will roll out and monitor these custom builds the same
way it does for the "pristine" CoreOS image today.

This is the OpenShift integration of [ostree native containers](https://fedoraproject.org/wiki/Changes/OstreeNativeContainer) or [CoreOS layering](coreos/enhancements#7) via the MCO.
cgwalters added a commit to cgwalters/enhancements that referenced this pull request Feb 10, 2022
**NOTE: Nothing in this proposal should be viewed as final.
It is highly likely that details will change.  It is quite possible
that larger architectural changes will be made as well.**

Change RHEL CoreOS as shipped in OpenShift to be a "base image" that can
be used as in layered container builds and then booted.  This will allow
custom 3rd party agents delivered via RPMs installed in a container
build.  The MCO will roll out and monitor these custom builds the same
way it does for the "pristine" CoreOS image today.

This is the OpenShift integration of [ostree native containers](https://fedoraproject.org/wiki/Changes/OstreeNativeContainer) or [CoreOS layering](coreos/enhancements#7) via the MCO.
cgwalters added a commit to cgwalters/enhancements that referenced this pull request Mar 8, 2022
**NOTE: Nothing in this proposal should be viewed as final.
It is highly likely that details will change.  It is quite possible
that larger architectural changes will be made as well.**

Change RHEL CoreOS as shipped in OpenShift to be a "base image" that can
be used as in layered container builds and then booted.  This will allow
custom 3rd party agents delivered via RPMs installed in a container
build.  The MCO will roll out and monitor these custom builds the same
way it does for the "pristine" CoreOS image today.

This is the OpenShift integration of [ostree native containers](https://fedoraproject.org/wiki/Changes/OstreeNativeContainer) or [CoreOS layering](coreos/enhancements#7) via the MCO.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants