bootc/rpm-ostree update fails on Importing: Unencapsulating base: failed to invoke method FinishPipe with zstd:chunked image

[travier]

Updating to a zstd:chunk'ed image using bootc/rpm-ostree fails with:

$ sudo bootc update
layers already present: 42; layers needed: 24 (1.3 GB)
Fetched layers: 0 B in 2 seconds (0 B/s)
ERROR Upgrading: Pulling: Importing: Unencapsulating base: failed to invoke method FinishPipe: failed to invoke method FinishPipe: write |1: broken pipe

$ sudo bootc status
No staged image present
Current booted image: quay.io/travier/fedora-kinoite:latest
    Image version: 41.20241207.0 (2024-12-07 04:10:35.798585027 UTC)
    Image digest: sha256:40efa2aa91e2ad4c3b75218cce8463d78193f2b35b2dfc972d412fba5eaaccb7
Current rollback image: quay.io/travier/fedora-kinoite:latest
    Image version: 41.20241202.0 (2024-12-02 04:11:16.290882830 UTC)
    Image digest: sha256:1807aefcb830c8952c1ecebaaab28900ef48ea5f8075e4a6b34a38d788f1da7e

$ sudo rpm-ostree update
Pulling manifest: ostree-image-signed:docker://quay.io/travier/fedora-kinoite:latest
Importing: ostree-image-signed:docker://quay.io/travier/fedora-kinoite:latest (digest: sha256:1f27336996fc663ad0ef3f448eef21edfa71322931fde0fe9f577068736aa2d5)
ostree chunk layers already present: 43
ostree chunk layers needed: 22 (1.2 GB)
custom layers needed: 1 (38.8 MB)
[0/23] Fetching ostree chunk f3c0f64a5cdf6196496 (50.4 MB)... done
error: Importing: Unencapsulating base: failed to invoke method FinishPipe: failed to invoke method FinishPipe: write |1: broken pipe

I had been using zstd:chunked images with success (https://github.com/travier/fedora-kinoite) until rpm-ostree 2024.9-1.fc41 landed in F41 recently. Turning the images compression back to zstd worked as a workaround which suggests that this is a zstd:chunk specific issue. This is very similar to #509 but only happened recently.

This also impacts bootc-1.1.2-2.fc41.x86_64 which suggests that this is a regression. It used to work with bootc but I don't remember the version it worked with.

• Last know good:
  • rpm-ostree 2024.8-1.fc41
• Know bad:
  • rpm-ostree 2024.9-1.fc41
  • bootc-1.1.2-2.fc41.x86_64

I haven't been able to pinpoint/bissect exactly which commit introduced this regression yet. Reporting this so that it does not get lost and I'll try to bissect the regression later.

[cgwalters]

This also impacts bootc-1.1.2-2.fc41.x86_64 which suggests that this is a regression

It may somehow be a regression rom ostreedev/ostree-rs-ext#677

[pmtk]

Hey, I believe we started observing this in our MicroShift CI for centos bootc images. Here's the issue: https://issues.redhat.com/browse/USHIFT-5546

[jeckersb]

This should be fixed once #1247 merges