From b8285e0d1932b84adb9a465ef29f1532dca91fe1 Mon Sep 17 00:00:00 2001 From: Harshal Patil Date: Mon, 18 Mar 2024 15:48:57 -0400 Subject: [PATCH] Lazy image pull support Signed-off-by: Harshal Patil --- enhancements/node/lazy-image-pull.md | 266 +++++++++++++++++++++++++++ 1 file changed, 266 insertions(+) create mode 100644 enhancements/node/lazy-image-pull.md diff --git a/enhancements/node/lazy-image-pull.md b/enhancements/node/lazy-image-pull.md new file mode 100644 index 0000000000..5a93794a4b --- /dev/null +++ b/enhancements/node/lazy-image-pull.md @@ -0,0 +1,266 @@ +--- +title: lazy-image-pull +authors: + - @harche +reviewers: # Include a comment about what domain expertise a reviewer is expected to bring and what area of the enhancement you expect them to focus on. For example: - "@networkguru, for networking aspects, please look at IP bootstrapping aspect" + - @mrunalp + - @rphillips + - @sairameshv + - @giuseppe + - @haircommander +approvers: # A single approver is preferred, the role of the approver is to raise important questions, help ensure the enhancement receives reviews from all applicable areas/SMEs, and determine when consensus is achieved such that the EP can move forward to implementation. Having multiple approvers makes it difficult to determine who is responsible for the actual approval. + - @mrunalp +api-approvers: # In case of new or modified APIs or API extensions (CRDs, aggregated apiservers, webhooks, finalizers). If there is no API change, use "None" + - @mrunalp +creation-date: 2024-03-18 +last-updated: 2024-03-18 +tracking-link: # link to the tracking ticket (for example: Jira Feature or Epic ticket) that corresponds to this enhancement + - https://issues.redhat.com/browse/OCPNODE-2204 +--- + +# Enable Lazy Image Pulling Support in OpenShift + + +## Summary + +This enhancement proposes integrating support for lazy image pulling in OpenShift using [Stargz Store plugin](https://github.com/containerd/stargz-snapshotter). This will provide the ability to lazy-pull compatible container images, significantly reducing container startup times for the specific workloads that can benefit from it. + +## Motivation + +* **Faster Serverless:** - Serverless end points need to serve the cold start requests as fast as possible. Lazy-pulling of the images allow the serverlesss end point to start serving the user request without getting blocked on the entire container image to get downloaded on the host. +* **Improved User Experience:** Large images often lead to extended pod startup times. Lazy-pulling drastically reduces the initial delay, providing a smoother user experience. +* **Potential Benefits for Large Images:** If the container image is shared between various containers which utilize only part of the image during their execution, Lazy image pulling speeds up such container startup by downloading only the relavent bits from those images. + +### User Stories + + +* As a cluster administrator, I want to serve serverless endpoints on my cluster as fast as possible by not waiting for the entire container image to download. +* As a developer, I want to deploy applications packaged in large container images while minimizing the impact of image size on pod startup time. + +# Goals + +* Add support for drop-in config in [container storage config](https://github.com/containers/storage/blob/main/pkg/config/config.go) +* Package upstream [Stargz Store plugin](https://github.com/containerd/stargz-snapshotter) for the installation on RHCOS +* Enable lazy image pulling in Openshift using [Stargz Store plugin](https://github.com/containerd/stargz-snapshotter) with [zstd:chunked](https://github.com/containers/storage/pull/775) image format. +* Support lazy image pulling only on the worker nodes (or subset of worker nodes). + +# Non-Goals + +* Enabling lazy image pulling on the control plane nodes. +* Replacement of the standard OCI image format; `zstd:chunked` is a variant of zstd compressed layers in OCI, which is supported by podman and moby +* Alterations to the image build process; developers can use standard OCI build tools, such as buildah to build and convert images to `zstd:chunked` format. +* Support for lazy image pull format other than `zstd:chunked` + +## Proposal + +In order to use `zstd:chunked` images, we need to have [Stargz Store plugin](https://github.com/containerd/stargz-snapshotter/blob/aaa46a75dd97e401025f82630c9d3d4e41c9f670/docs/INSTALL.md#install-stargz-store-for-cri-opodman-with-systemd) installed on the node and with a configuration for container storage library. [ContainerRuntimeConfig](https://github.com/openshift/machine-config-operator/blob/release-4.16/docs/ContainerRuntimeConfigDesign.md) will be used to enable this feature in OCP. + +### Workflow Description + +* A cluster administrator will create or edit the [ContainerRuntimeConfig](https://github.com/openshift/machine-config-operator/blob/release-4.16/docs/ContainerRuntimeConfigDesign.md) object to enable lazy image pulling support by, +```yaml +apiVersion: machineconfiguration.openshift.io/v1 +kind: ContainerRuntimeConfig +metadata: + name: lazy-image-pull +spec: + machineConfigPoolSelector: + matchLabels: + pools.operator.machineconfiguration.openshift.io/worker: "" + containerRuntimeConfig: + lazyImagePull: + enabled: true +``` +* Optionally, A list of supported types of the image format for lazy pulling can also be specified. At the time of writing this document, Openshift will only support `zstd:chunked` type image format. If the user doesn't explicitly specify the image format, the default value is set to `zstd:chunked`. +```yaml +apiVersion: machineconfiguration.openshift.io/v1 +kind: ContainerRuntimeConfig +metadata: + name: lazy-image-pull +spec: + machineConfigPoolSelector: + matchLabels: + pools.operator.machineconfiguration.openshift.io/worker: "" + containerRuntimeConfig: + lazyImagePull: + enabled: true + format: + - "zstd:chunked" +``` +### API Extensions + +Extend existing [ContainerRuntimeConfiguration](https://github.com/openshift/api/blob/ff84c2c732279b16baccf08c7dfc9ff8719c4807/machineconfiguration/v1/types.go#L749C6-L749C35) to hold the required configuration for lazy image pulling. + +```golang +// ContainerRuntimeConfiguration defines the tuneables of the container storage +type ContainerRuntimeConfiguration struct { + // +kubebuilder:validation:Optional + LazyImagePullConfig *LazyImagePullConfiguration `json:"lazyImagePullConfig, omitempty"` +} +``` + +```golang +type LazyImagePullConfiguration struct { + // +kubebuilder:validation:Required + Enable bool `json:"enable"` + + // +kubebuilder:validation:Optional + Format []string `json:"format, omitempty"` +} +``` + +### Topology Considerations + +#### Hypershift / Hosted Control Planes + +N/A + +#### Standalone Clusters + +N/A + +#### Single-node Deployments or MicroShift + +N/A + +### Implementation Details/Notes/Constraints + +## Package Stargz Store plugin + +Package the Stargz Store plugin located at, https://github.com/containerd/stargz-snapshotter so that it can get installed on RHCOS + +## Drop-in config support for c/storage +A configuration for `additionallayerstores` needs to be added in `/etc/containers/storage.conf.d/lazy-image-pull.conf` on the node as follows, +```toml= +[storage.options] +additionallayerstores = ["/var/lib/stargz-store/store:ref"] +``` +As of now, that `c/storage` configuration file does not support drop-in config. The entire content of that file is [hard-coded in MCO](https://github.com/openshift/machine-config-operator/blob/release-4.16/templates/common/_base/files/container-storage.yaml). Without the drop-in config support, we risk overriding that critical file and potentially losing custom configuration that might be already present in there. A Request for adding drop-in config support for applying similar custom configurations to `c/storage` [already exists](https://github.com/containers/storage/issues/1306). Hence, this proposal advocates adding support for drop-in config in `c/storage`. + + + +## Starz Store systemd service + +Startz Store service needs to be running on the host _before_ crio service starts. MCO needs to drop relavent systemd unit file on the targeted nodes. + +```toml= +[Unit] +Description=Stargz Store plugin +Before=crio.service + +[Service] +Type=notify +ExecStart=/usr/bin/stargz-store --log-level=debug --config=/etc/stargz-store/config.toml /var/lib/stargz-store/store +ExecStopPost=umount /var/lib/stargz-store/store +Restart=always +RestartSec=1 + +[Install] +WantedBy=crio.service +``` + +### Risks and Mitigations + +## Enhancing c/storage Configuration Management + +Even in the absence of drop-in configuration support within `c/storage`, the current [hard-coded](https://github.com/openshift/machine-config-operator/blob/release-4.16/templates/common/_base/files/container-storage.yaml) `/etc/containers/storage.conf` file can be updated by the MCO. However, the integration of drop-in configuration support would offer a more secure and robust method for managing this configuration file. Should the drop-in configuration feature not be implemented in `c/storage`, the MCO has the capability to read the existing, hard-coded `storage.conf` and appropriately inject the necessary `additionallayerstores` settings. + +### Drawbacks + +#### Resource Consumption with FUSE + +The stargz-snapshotter employs a FUSE filesystem, which can lead to increased resource usage on the node during intensive local disk I/O operations. It’s important to note that this does not affect containers performing heavy I/O with local volume mounts, as they will not experience performance degradation. Although in near future this might get mitigated by [FUSE-Passthrough](https://www.phoronix.com/news/Linux-6.9-FUSE-Passthrough) + + +## Test Plan + +#### OpenShift End-to-End Testing + +Write comprehensive e2e tests to confirm the accurate pulling and execution of containers utilizing zstd:chunked images. + +#### Performance Benchmarking + +Benchmarks comparing the performance of zstd:chunked images against traditional image usage across diverse operational scenarios. + + +## Graduation Criteria +#### Dev Preview +* Introduce support for lazy image pulling per machine config pool in Openshift + +## Upgrade / Downgrade Strategy + +* Downgrade expectations - In the event of a cluster downgrade to a version that does not support zstd:chunked image format, containers utilizing such image format will get downloaded again and their startup times would be comparable to the non-zstd:chunked equivalent container image. Even in the absence of stargz-snapshotter, those container will start but without any performance gains. + +## Version Skew Strategy + +#### During an upgrade, we will always have skew among components, how will this impact your work? + +During the upgrade, if the container that uses zstd:chunked image format gets scheduled on the node that doesn't yet support it, the container will start without any performance gains associated with zstd:chunked image format. + +#### Does this enhancement involve coordinating behavior in the control plane and in the kubelet? + +No + +#### How does an n-2 kubelet without this feature available behave when this feature is used? + +If the n-2 kubelet is running on the host without the required `additionallayerstores` configuration for container storage library then the container that use zstd:chunked image format will start without any associated performance gains. + +#### Will any other components on the node change? + +Yes, configuration for the `c/storage` library used by CRI-O. + +## Operational Aspects of API Extensions + +#### For conversion/admission webhooks and aggregated apiservers: what are the SLIs (Service Level Indicators) an administrator or support can use to determine the health of the API extensions + +metrics `kubelet_pod_start_sli_duration_seconds`, `kubelet_pod_start_duration_seconds` abd `kubelet_runtime_operations_duration_seconds` can be used to gauge the performance of the container start up time. + +#### What impact do these API extensions have on existing SLIs (e.g. scalability, API throughput, API availability) + +Container that use large images but initiate the execution with a relatively small binary (e.g. using `ping` from fedora:39 container image) in that image should see faster start up time. + +#### How is the impact on existing SLIs to be measured and when (e.g. every release by QE, or automatically in CI) and by whom (e.g. perf team; name the responsible person and let them review this enhancement) + +Container start up time can be measured by an e2e CI test to validate the expected speed up. + +#### Describe the possible failure modes of the API extensions. + +MCO could encounter failures in rolling out the required machine config to enable this feature. + +#### Describe how a failure or behaviour of the extension will impact the overall cluster health (e.g. which kube-controller-manager functionality will stop working), especially regarding stability, availability, performance and security. + +If MCO fails to roll out required machine config to enable this feature, the container images that use `zstd:chunked` format will not observe improved start up times associated with their `zstd:chunked` image. + +#### Describe which OCP teams are likely to be called upon in case of escalation with one of the failure modes and add them as reviewers to this enhancement. + +OCP Node Team + +## Support Procedures + +### Detect the failure modes in a support situation, describe possible symptoms (events, metrics, alerts, which log output in which component) + +* If MCO fails to roll out required machine config to enable this feature, the container images that use zstd:chunked format will not observe any associated performance gains in their start up time. +* If the workload is going to do heavy local disk IO, then this feature may contribute to higher resource memory and cpu usage on the node due to the use of FUSE filesystem. +* Useful metrics to observe container start up time, `kubelet_pod_start_sli_duration_seconds`, `kubelet_pod_start_duration_seconds` abd `kubelet_runtime_operations_duration_seconds`. + +### Disable the API extension (e.g. remove MutatingWebhookConfiguration xyz, remove APIService foo) + +#### What consequences does it have on the cluster health? + +Containers that use zstd:chunked image format will observe no associated performance gains in their start up time. + +#### What consequences does it have on existing, running workloads? + +MCO will remove the required configuration in the container storage library on the host, which would result in container images using zstd:chunked format getting downloaded like regular OCI container images. + +#### What consequences does it have for newly created workloads? + +MCO will remove the required configuration in the container storage library on the host, which would result in container images using zstd:chunked format getting downloaded like regular OCI container images. + +### Does functionality fail gracefully and will work resume when re-enabled without risking consistency? + +Yes + +## Alternatives + +1. [Seekable OCI images](https://aws.amazon.com/blogs/containers/under-the-hood-lazy-loading-container-images-with-seekable-oci-and-aws-fargate/) - Only works with [containerd](https://github.com/awslabs/soci-snapshotter/blob/main/docs/install.md).