Support for split layers #123
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cgwalters
force-pushed
the
tar-split
branch
3 times, most recently
from
9d256af
to
d90a1b1
Compare
cgwalters
force-pushed
the
tar-split
branch
3 times, most recently
from
a88589c
to
506d2bd
Compare
cgwalters
force-pushed
the
tar-split
branch
2 times, most recently
from
531715f
to
8ce5b41
Compare
|
One thing that's clear to me now is that we will need to inject metadata to differentiate between the layers comprising the "ostree base" and all the remaining layers. Because in order to perform SELinux labeling on derived content, we need to know the base. Alternatively, we just go for #159 which obviates this problem. I am leaning in that direction more and more strongly. |
|
If we don't do #159 then instead what I am now realizing is that because docker schema 2 doesn't have any "annotations" unlike oci we'd need to stick that data on the config object. Which in turn blocks on containers/containers-image-proxy-rs#14 |
cgwalters
force-pushed
the
tar-split
branch
2 times, most recently
from
2feec21
to
9525086
Compare
cgwalters
force-pushed
the
tar-split
branch
11 times, most recently
from
4745d93
to
fa957bf
Compare
|
OK, this is finally in a state where it seems to work! Lifting draft, but it's a huge change so we need careful review and thought. At least the core logic today is only opt-in, you have to |
cgwalters
force-pushed
the
tar-split
branch
5 times, most recently
from
8e86c9a
to
4fe97f1
Compare
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 21, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 22, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 23, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 23, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 23, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
Closes: ostreedev#69 This is initial basic support for splitting files (objects) from a commit into separate container image layers, and reassembling those layers into a commit on the client. We retain our present logic around e.g. GPG signature verification. There's a new `chunking.rs` file which has logic to automatically factor out things like the kernel/initramfs and large files. In order to fetch these images client side, we now heavily intermix/cross the previous code for fetching non-ostree layers.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 23, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
|
OK, CI on coreos/rpm-ostree#3478 is passing! I'd like to merge this now, I've fixed the bugs that caused it break the current v0 container format, so while it's a lot of new code, I think the risk is pretty low because most of the new bits are opt-in. |
jlebon
approved these changes
Mar 24, 2022
jmarrero
approved these changes
Mar 24, 2022
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 24, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Mar 29, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/ostree-rs-ext
that referenced
this pull request
Apr 4, 2022
The biggest thing here was the merge of "chunked ostree" support in ostreedev#123 cascading into various API changes. But I think it's working, let's cut 0.7 so the next rpm-ostree can roll in the support.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Apr 4, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Apr 4, 2022
Depends ostreedev/ostree-rs-ext#123 We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to cgwalters/rpm-ostree
that referenced
this pull request
Apr 4, 2022
The headlining feature here is ostreedev/ostree-rs-ext#123 which adds support for "chunked ostree". We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
cgwalters
added a commit
to coreos/rpm-ostree
that referenced
this pull request
Apr 6, 2022
The headlining feature here is ostreedev/ostree-rs-ext#123 which adds support for "chunked ostree". We gain a new `rpm-ostree container-encapsulate` option which is like `ostree container encapsulate`, but generates chunked images using the RPM database. And on the client side, we now know how to handle incremental updates - chunks that haven't changed won't be redownloaded.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Support for split layers
Closes: #69
This is initial basic support for splitting files (objects) from
a commit into separate container image layers, and reassembling
those layers into a commit on the client.
We retain our present logic around e.g. GPG signature verification.
There's a new
chunking.rsfile which has logic to automaticallyfactor out things like the kernel/initramfs and large files.
In order to fetch these images client side, we now heavily
intermix/cross the previous code for fetching non-ostree layers.