Initial installs from c/storage require full fetch from registry subsequently

[cgwalters]

This is a messy topic. It all relates to the giant mistake that docker did originally (and was carried forward into OCI) of checksumming compressed data.

Basically when one does a podman run <image> bootc install to-filesystem this will default to operating on an uncompressed c/storage representation; when we ask c/storage to give us a manifest, it synthesizes an uncompressed one.

This, combined with the fact that the ostree-container store only caches based on blob digests means that when one does an install today through our default/expected flow, and then you want to switch to a registry you end up needing to re-download all the layers at first (because of the switch from uncompressed -> compressed).

However subsequent updates will always operate on the blob digests from the registry which stays compressed.

---

Related issues came up in the past here that c/storage doesn't offer a way (at least by default?) to access the real original manifest (with the compressed layers). See containers/image#2599 (and containers/storage#1849 ).

But, I think we can and should fix the ostree-container storage here to also track layers via diffid - when we go to do a fetch, map from the uncompressed layers to the diffids from the manifest we're going to fetch, then we can skip downloading those etc.

[bdherouville]

This issue is blocking for far-edge usecase where the 4G data plan can be burned by only 5/10 devices. The current workaround is to deploy the bootc image filesystem to a device, then extract an image from this device after it fetched the registry. Can we have a workaround by reinjecting the registry into the image after we built it ?

[jjaswanson4]

Agree with @bdherouville, and just adding my $0.02: ostree uses a concept of static deltas to greatly reduce the amount of work and data the remote deployments need to do to move to the next image: https://sigs.centos.org/automotive/building/creating_static_deltas/

This costs some additional CPU cycles wherever the ostree repo is, however that's a small price to pay when running remote systems over 4G/5G or even satellite links.

I think this is a great idea:

But, I think we can and should fix the ostree-container storage here to also track layers via diffid - when we go to do a fetch, map from the uncompressed layers to the diffids from the manifest we're going to fetch, then we can skip downloading those etc.

But I also support whatever solution aligns with "the remote systems only need to pull the difference between where they are, and where they're going", loosely speaking.

[cgwalters]

ostree uses a concept of static deltas to greatly reduce the amount of work and data the remote deployments need to do to move to the next image

Hi, yes; I created that. But the real thing here is that for core ostree (apart from static deltas) the object checksums always refer to uncompressed representation, so there is no loss or confusion when moving from "stored remotely, compressed" to "unpacked on device". ostree's signatures especially cover the uncompressed commit (as opposed to current e.g. sigstore covering the manifest which covers compressed blobs for OCI; xref opencontainers/distribution-spec#235 )

(As far as static deltas, see also containers/image#902 (comment) )

The current workaround is to deploy the bootc image filesystem to a device, then extract an image from this device after it fetched the registry. Can we have a workaround by reinjecting the registry into the image after we built it ?

One workaround today is to always install from a registry, which is how the default Anaconda (PXE) flow works today (but not how bootc-image-builder or bootc install to-filesystem work). Although, this points out that there is a potentially different and simpler fix for the bootc install to-filesystem path. Today we have a "fetch check" that sanity checks that we can find a manifest for the target image - but we just throw that away.

What we really should do is fetch that manifest instead and use it - but treat the image in containers-storage: as a cache (mapped via diffids).

So basically special case just the install path, instead of special casing every fetch to support indirecting via diffids.