bootc image copy-to-storage broken since skopeo v1.18.0

[cgwalters]

ref https://artifacts.dev.testing-farm.io/3e2ef318-cfae-486c-a819-82939d53e452/

              out: ERROR Pushing image: Export: Skopeo copy: skopeo failed: time="2025-02-20T20:08:40Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
                out: time="2025-02-20T20:09:04Z" level=fatal msg="writing blob: layer 65 (blob \"sha256:ce440240c7b6581fdea4df75005a178507325ee0055d3a5ba62778157a151c61\"/\"\"/\"sha256:7922ec7f545705c4bcb9d1f85073f3ea7a523feb5d6c6e27cde163b8e3666b9b\") does not match config's DiffID \"sha256:e925d37afbb392200d6df4a72eb9292779fd9d0429f05ee960f6cc5b1c36ba5a\""
                out: 

I suspect this may relate to recent c/storage or c/image changes like

• chunked: store UncompressedDigest if validated containers/storage#1813

[cgwalters]

I glanced a bit more on this, I think this is coming from bootc image copy-to-storage. Currently, the ostree-container store does not have a tar-split/splitstream equivalent, and it's quite possible here that indeed the config.json we are synthesizing is exactly the upstream one and so we've always been emitting a broken diffid (because we're using a totally different Rust tar writer and it's very unlikely it happens to match what the Go writer does), and it's just that recent skopeo (c/storage, c/image) is actually correctly validating this and failing.

If that's the case, then this will be a bit ugly to fix as what we'll need to do is switch to synthesizing tarballs and then updating the config.json from those.

---

Currently, bootc image copy-to-storage is an experimental interface, so this failing should just be only impacting our own CI.

---

Also of note, it looks like https://github.com/containers/skopeo/releases/tag/v1.18.0 was very recently released and landed in :stream9 so that tracks.

[cgwalters]

(Also it'd sure be nice to have the equivalent of bash set -x for our nushell code, or maybe at least something that verbosely prints selected commands)

[cgwalters]

and it's quite possible here that indeed the config.json we are synthesizing is exactly the upstream one and so we've always been emitting a broken diffid (

But we have code to explicitly handle this...

https://github.com/containers/bootc/blob/5309d5b31b57027df917c97271f628f28ed59b2b/ostree-ext/src/container/store.rs#L1410-L1412

[cgwalters]

Also of note, it looks like https://github.com/containers/skopeo/releases/tag/v1.18.0 was very recently released and landed in :stream9 so that tracks.

The error message comes from containers/image@2e79eab which further correlates here as the previous skopeo release was from November, so we just got that commit.

[cgwalters]

Basically @jeckersb I'd check if the output OCI is valid or not objectively; probably add --transport=oci to copy-to-storage so it's easier to inspect before we then use skopeo to copy it to c/storage.

It is perhaps possible there's a bug in https://github.com/containers/ocidir-rs/ that allows us to generate incorrect diffids (uncompressed tar streams). Or it's possible there's a bug in that commit to c/image when interacting with oci directories (as opposed to registries?).

[mh21]

@cgwalters:

Currently, bootc image copy-to-storage is an experimental interface, so this failing should just be only impacting our own CI.

QE engineers have been told that this interface is the preferred way to rebuild the container image on the host itself, so it is actually impacting QE teams trying to come to grips with image mode 😂

[cgwalters]

@jeckersb can you backport this patch? Or maybe we should just cut a new release today and ship that?

[cgwalters]

QE engineers have been told that this interface is the preferred way to rebuild the container image on the host itself

Yes and to be clear eventually we will make this more ergonomic, and we do test it, the gap is that we don't run those tests as part of our integration tests for the container image. (Which would have blocked the updated skopeo)

I understand why this flow is in use. However I do want to emphasize that the primary target use case for bootc today is building a container on a remote host. We definitely will support the on-host-build flow much better in the future.