Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to ostree-ext 0.7, add new container-encapsulate #3478

Merged
merged 1 commit into from
Apr 6, 2022

Conversation

cgwalters
Copy link
Member

@cgwalters cgwalters commented Feb 28, 2022

The headlining feature here is ostreedev/ostree-rs-ext#123
which adds support for "chunked ostree".

We gain a new rpm-ostree container-encapsulate option which
is like ostree container encapsulate, but generates chunked
images using the RPM database.

And on the client side, we now know how to handle incremental
updates - chunks that haven't changed won't be redownloaded.


@openshift-ci
Copy link

openshift-ci bot commented Feb 28, 2022

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@cgwalters
Copy link
Member Author

With this try e.g. ./rpm-ostree ex-dump-content-mapping --repo=/var/srv/walters/builds/fcos/tmp/repo --ref fedora/coreos/x86_64/testing-devel

@cgwalters
Copy link
Member Author

cgwalters commented Mar 3, 2022

Demos:

FCOS:

$ rpm-ostree container-encapsulate --repo=tmp/repo --ref=fedora/x86_64/coreos/testing-devel oci:tmp/fcos.oci
18958 objects in 407 packages (272 source)
rpm size: 1337124640
Earliest changed package: shim-x64-15.4-5.x86_64 at 2021-05-05 01:07:03 UTC
1004 duplicates
Multiple owners:
  /usr/lib/.build-id/35/ab2e40ddafa770aa0bc3bd883d5339b2deff03
  /usr/lib/.build-id/f9/12ca8c9b1b274dfddb5e3fe6ab6c28d6c5e29c
  /usr/sbin/ip6tables
  /usr/sbin/ip6tables-restore
  /usr/sbin/ip6tables-save
  /usr/sbin/iptables
  /usr/sbin/iptables-restore
  /usr/sbin/iptables-save
  /usr/share/licenses/systemd/LICENSE.LGPL2.1
Pushed digest: sha256:42ced2fad7587fc3b6663d2a1622a0c651b1cf12cc6d77ac4364428cc7c91972

I've uploaded that to quay.io/cgwalters/fcos-chunked

@cgwalters cgwalters force-pushed the update-ostree-ext-layered branch 3 times, most recently from dc76688 to 993418d Compare March 4, 2022 22:10
@cgwalters cgwalters force-pushed the update-ostree-ext-layered branch 2 times, most recently from c0ff985 to b06b741 Compare March 12, 2022 19:44
@cgwalters cgwalters force-pushed the update-ostree-ext-layered branch 3 times, most recently from d084970 to 0893ca4 Compare March 23, 2022 13:18
@cgwalters
Copy link
Member Author

/test all

@cgwalters
Copy link
Member Author

/test all

1 similar comment
@cgwalters
Copy link
Member Author

/test all

@cgwalters
Copy link
Member Author

/test all

@cgwalters cgwalters force-pushed the update-ostree-ext-layered branch 2 times, most recently from 25f1507 to 85f8727 Compare March 29, 2022 19:53
cgwalters added a commit to cgwalters/coreos-assembler that referenced this pull request Mar 30, 2022
This *opts in* to using coreos/rpm-ostree#3478
so we can safely ratchet in the change, and more easily test in
in non-production CI configurations but still using coreos-assembler.
cgwalters added a commit to cgwalters/coreos-assembler that referenced this pull request Mar 30, 2022
This *opts in* to using coreos/rpm-ostree#3478
so we can safely ratchet in the change, and more easily test in
in non-production CI configurations but still using coreos-assembler.
cgwalters added a commit to cgwalters/coreos-assembler that referenced this pull request Mar 31, 2022
This *opts in* to using coreos/rpm-ostree#3478
so we can safely ratchet in the change, and more easily test in
in non-production CI configurations but still using coreos-assembler.
cgwalters added a commit to cgwalters/coreos-assembler that referenced this pull request Mar 31, 2022
This *opts in* to using coreos/rpm-ostree#3478
so we can safely ratchet in the change, and more easily test in
in non-production CI configurations but still using coreos-assembler.
jlebon pushed a commit to coreos/coreos-assembler that referenced this pull request Apr 1, 2022
This *opts in* to using coreos/rpm-ostree#3478
so we can safely ratchet in the change, and more easily test in
in non-production CI configurations but still using coreos-assembler.
cgwalters added a commit to cgwalters/coreos-assembler that referenced this pull request Apr 4, 2022
This requires ostreedev/ostree-rs-ext#275

When `deploy-via-container: true` is used in combination with
the new `rpm-ostree container-encapsulate` from
coreos/rpm-ostree#3478

We need to use the ostree-container synthetic merge commit to find
the deployment.

(Actually, what we want is something more like `ostree admin status --json`,
 or I guess for now `rpm-ostree status --json` to work offline; but
 for now this is good too)
@cgwalters cgwalters force-pushed the update-ostree-ext-layered branch 2 times, most recently from 69617ec to 250e68e Compare April 4, 2022 18:38
jlebon pushed a commit to coreos/coreos-assembler that referenced this pull request Apr 4, 2022
This requires ostreedev/ostree-rs-ext#275

When `deploy-via-container: true` is used in combination with
the new `rpm-ostree container-encapsulate` from
coreos/rpm-ostree#3478

We need to use the ostree-container synthetic merge commit to find
the deployment.

(Actually, what we want is something more like `ostree admin status --json`,
 or I guess for now `rpm-ostree status --json` to work offline; but
 for now this is good too)
The headlining feature here is ostreedev/ostree-rs-ext#123
which adds support for "chunked ostree".

We gain a new `rpm-ostree container-encapsulate` option which
is like `ostree container encapsulate`, but generates chunked
images using the RPM database.

And on the client side, we now know how to handle incremental
updates - chunks that haven't changed won't be redownloaded.
@cgwalters cgwalters changed the title WIP: Adapt to and use new ostree-ext tar-split branch Update to ostree-ext 0.7, add new container-encapsulate Apr 4, 2022
@cgwalters cgwalters marked this pull request as ready for review April 4, 2022 22:58
@cgwalters
Copy link
Member Author

                                               #0  0x00007f603962988c __pthread_kill_implementation (libc.so.6 + 0x8f88c)
                                               #1  0x00007f60395dc6a6 raise (libc.so.6 + 0x426a6)
                                               #2  0x00007f60395c67d3 abort (libc.so.6 + 0x2c7d3)
                                               #3  0x00005636a1c4b027 _ZN11panic_abort18__rust_start_panic5abort17heb2307cc81785ad5E.llvm.14848102273724856416 (rpm-ostree + 0x2ef027)
                                               #4  0x00005636a2083a36 rust_panic (rpm-ostree + 0x727a36)
                                               #5  0x00005636a2083918 _ZN3std9panicking20rust_panic_with_hook17hffe17de889022375E (rpm-ostree + 0x727918)
                                               #6  0x00005636a2082fdf _ZN3std9panicking19begin_panic_handler28_$u7b$$u7b$closure$u7d$$u7d$17h9df7a4eeedd84d36E (rpm-ostree + 0x726fdf)
                                               #7  0x00005636a20820d4 _ZN3std10sys_common9backtrace26__rust_end_short_backtrace17h50480516fb3ba642E.llvm.7442842944024386991 (rpm-ostree + 0x7260d4)
                                               #8  0x00005636a2082f5d rust_begin_unwind (rpm-ostree + 0x726f5d)
                                               #9  0x00005636a1a51b41 _ZN4core9panicking9panic_fmt17h92d66d0ceae7016cE (rpm-ostree + 0xf5b41)
                                               #10 0x00005636a20c1408 _ZN4core9panicking13panic_display17hc2cfe6d608e32bc9E.llvm.16789651492748722405 (rpm-ostree + 0x765408)
                                               #11 0x00005636a1ea4e98 rpmostreecxx$cxxbridge1$tokio_handle_get (rpm-ostree + 0x548e98)
                                               #12 0x00005636a20fc8ea _ZN12rpmostreecxx16tokio_handle_getEv (rpm-ostree + 0x7a08ea)
                                               #13 0x00005636a2118661 _ZL27rpmostreed_transaction_initP22_RpmostreedTransaction (rpm-ostree + 0x7bc661)
                                               #14 0x00007f603a2ae34b g_type_create_instance (libgobject-2.0.so.0 + 0x3b34b)
                                               #15 0x00007f603a295d4d g_object_new_internal (libgobject-2.0.so.0 + 0x22d4d)
                                               #16 0x00007f603a2972c8 g_object_new_valist (libgobject-2.0.so.0 + 0x242c8)
                                               #17 0x00007f603a34921e g_initable_new_valist (libgio-2.0.so.0 + 0x7a21e)
                                               #18 0x00007f603a3492ed g_initable_new (libgio-2.0.so.0 + 0x7a2ed)
                                               #19 0x00005636a211d5dc rpmostreed_transaction_new_cleanup (rpm-ostree + 0x7c15dc)
                                               #20 0x00005636a212311c _ZL17os_handle_cleanupP12_RPMOSTreeOSP22_GDBusMethodInvocationPKPKc (rpm-ostree + 0x7c711c)
                                               #21 0x00007f6038fd4c04 ffi_call_unix64 (libffi.so.6 + 0x6c04)
                                               #22 0x00007f6038fd4108 ffi_call (libffi.so.6 + 0x6108)
                                               #23 0x00007f603a28cbfd g_cclosure_marshal_generic (libgobject-2.0.so.0 + 0x19bfd)
                                               #24 0x00007f603a286c7f g_closure_invoke (libgobject-2.0.so.0 + 0x13c7f)
                                               #25 0x00007f603a2a2b14 signal_emit_unlocked_R (libgobject-2.0.so.0 + 0x2fb14)
                                               #26 0x00005636a21502ec _rpmostree_os_skeleton_handle_method_call (rpm-ostree + 0x7f42ec)
                                               #27 0x00007f603a3ed5c9 dispatch_invoke_in_context_func (libgio-2.0.so.0 + 0x11e5c9)
                                               #28 0x00007f603a18c9d5 g_main_context_invoke_full (libglib-2.0.so.0 + 0x549d5)
                                               #29 0x00007f603a3f2461 dispatch_in_thread_func (libgio-2.0.so.0 + 0x123461)
                                               #30 0x00007f603a38050f g_task_thread_pool_thread (libgio-2.0.so.0 + 0xb150f)
                                               #31 0x00007f603a1ba094 g_thread_pool_thread_proxy.lto_priv.0 (libglib-2.0.so.0 + 0x82094)
                                               #32 0x00007f603a1b77c2 g_thread_proxy (libglib-2.0.so.0 + 0x7f7c2)
                                               #33 0x00007f6039627b1a start_thread (libc.so.6 + 0x8db1a)
                                               #34 0x00007f60396ac660 __clone3 (libc.so.6 + 0x112660)
                                               

Hmmmm. This stack trace implies to me the tokio handle code threading has been buggy for quite a while. But I don't understand why it's not always crashing.

@cgwalters
Copy link
Member Author

/retest

@cgwalters
Copy link
Member Author

OK this one died in the compose tests, but it can't really break those. I'd like to get this in for the next release, any review takers?

@cgwalters
Copy link
Member Author

/override continuous-integration/jenkins/pr-merge

@openshift-ci
Copy link

openshift-ci bot commented Apr 5, 2022

@cgwalters: Overrode contexts on behalf of cgwalters: continuous-integration/jenkins/pr-merge

In response to this:

/override continuous-integration/jenkins/pr-merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Copy link
Member

@jlebon jlebon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't review this in depth, but it's all just in the container path which makes this less risky. So LGTM overall!

Still, would be great to have tests for this since it's a lot of non-trivial logic.

@@ -954,6 +954,7 @@ get_refts_for_rootfs (const char *rootfs, GLnxTmpDir *tmpdir)
gboolean
rpmostree_get_refts_for_commit (OstreeRepo *repo, const char *ref, RpmOstreeRefTs **out_ts,
GCancellable *cancellable, GError **error)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor/optional: extraneous blank line.

Ok(())
}

fn gv_nevra_to_string(pkg: &glib::Variant) -> String {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pretty sure we have this already on the C side. Might be worth making this public and moving over those callers.

@cgwalters
Copy link
Member Author

Definitely re CI...I'm trying to figure out too how to actually share that across repos. But CI does cover the not-split case.

@cgwalters cgwalters merged commit e61f8df into coreos:main Apr 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants