derivation: Support importing commit metadata, or original commit

[cgwalters]

Today rpm-ostree injects a lot of commit metadata, including things like:

• hash of input state
• input rpm-md repos and their timestamps
• rpmdb
• The new ostree.linux key with the kernel version

Right now we toss all of that when deploying via the new layered container image flow - even if that container has just a single layer. The reason for that is that we need to store e.g. the container image manifest.

There are a few potential directions to go:

Map ostree metadata to oci annotations

In this we map commit metadata to oci annotations. A major benefit of this is things like ostree.linux (i.e. kernel version) would also be immediately visible in skopeo inspect and similar tools.

Actually this path leads towards the obvious thing of also exporting the rpmdb for container images we generate that are built via yum. (Nontrivial work there)

Store container metadata out of band

Perhaps instead of synthesizing a new commit object and dropping the original encapsulated commit on the floor, we should instead write the original commit as is, but store the manifest as e.g. local detached metadata on the commit? Maybe use extended attributes?

Or store it as a commit, but in a new ref like /ostree/containers/manifest/<digest> and also retain the original commit?

[cgwalters]

Splitting this out of discussion here coreos/coreos-assembler#2523 (comment)

[jlebon]

Store container metadata out of band

Perhaps instead of synthesizing a new commit object and dropping the original encapsulated commit on the floor, we should instead write the original commit as is, but store the manifest as e.g. local detached metadata on the commit? Maybe use extended attributes?

Or store it as a commit, but in a new ref like /ostree/containers/manifest/<digest> and also retain the original commit?

Huge +1 on conserving the original commit, and deploying it too, so that rpm-ostree status matches. I think it'll make things much less confusing if it matches between the two worlds and preserving commit metadata and the rpm-ostree status --json API. Maybe we can add a ContainerDigest: field showing the container SHA256?

[cgwalters]

The core tension here is that in the current derivation model, all the metadata we have is from the base image - unless we require a finalization step like RUN ostree container finalize to be run which would create a commit object.

I've been going back and forth a lot on whether this should be required or not. There are a lot of useful things we can do if it is required, including doing basic linting and analysis. (In general, we can move things like all the logic in rpm-ostree to translate /var into tmpfiles.d into ostree core, etc.)

Well, all of that I guess is related, but still doesn't solve the problem that the manifest necessarily lives outside of the commit and we want to store it (and the config, which we are currently omitting).

Anyways, I think it's clear we should write a ref for the base commit, we are currently requiring that we have "ostree exported base" (though in theory, that could change).

But the core issue here is the minute that we allow derivation via containers, all of the base commit metadata (such as the rpmdb and ostree.linux) becomes potentially invalid.

[cgwalters]

PR in #147