Release Proposal v2.25

[mzaian]

This proposal is for the release of v2.25.0.

Deprecation / Removal

• Remove support for Kubernetes 1.26.x (move min version to 1.27.x) (update kube-version-min-required to v1.27 #10817, @KubeKyrie)
• Remove documentation for removed in-tree openstack provider (Remove documentation for removed in-tree openstack provider #10889, @LarssonOliver)

Feature / Major Changes

• A check is introduced to fail the playbook if cgroups are not enabled on the node (feat: Adding a check which determines if cgroups are enabled on a nod… #11165, @franznemeth)
• Add Calico v3.27.3 and make it default (Add Calico v3.27.3 and make it default #11141, @pomland-94)
• Add extra_vars support to vagrant setup (Add extra_vars support to vagrant setup #10932, @VannTen)
• Add kube-vip LeaderElection variables vip_leaseduration, vip_renewdeadline, vip_retryperiod options for kube-vip (make kube-vip LeaderElection variables configurable #11021, @KubeKyrie)
• Add new option remove_anonymous_access to prevent granting RBAC permissions to anonymous users. (Remove access to cluster from anonymous users #11016, @nicolas-goudry)
• Add scheduler plugins support (scheduler_plugins_enabled enable or disable the installation scheduler plugins / scheduler_plugins_enabled_plugins describe the enabled plugins / scheduler_plugins_diabled_plugins describe the disabled plugins / scheduler_plugins_plugin_config set the custom config for enabled plugins) (Add scheduler plugins support #10747, @tu1h)
• Added a config option to filter ntp interfaces (ntp: add config to filter and set ntp interfaces #11066, @Pavan-Gunda)
• Adding egress IPv6 for node-local-dns queries (k8s_allowed_egress_ipv6_ips) (Adding egress/ingress IPv6 for node-local-dns queries #10396, @raviranjanelastisys)
• Bump docker version for kylin linux (bump docker version for kylin linux #11203, @ErikJiang)
• Bump docker version for openeuler linux (bump docker version for openeuler linux #11206, @ErikJiang)
• Update almalinux-8 base image to 8.9 (Update almalinux-8 base image to 8.9 #10918, @VannTen)
• Bumping checksums and various versions (kubespray-defaults: regenerate checksums and bump various versions #10999, @MrFreezeex)
• Containerd: allow to configure fallback server (containerd: allow to configure fallback server #10988, @sathieu)
• Docker upgrade from 24.0 to 26.1 (Bump docker version from 24.0 to 26.1 #11198, @tico88612)
• Download hash script: auto discover versions (Download hash script: auto discover versions #10849, @VannTen)
• Enable configuring mountOptions, reclaimPolicy and volumeBindingMode for cinder-csi StorageClasses. (Enable configuring mountOptions, reclaimPolicy and volumeBindingMode … #10450, @Payback159)
• Make containerd v1.7.15 default (set default containerd_version to v1.7.15 and add checksums #11083, @Payback159)
• Make kubernetes v1.28.6 default ([kubernetes] Make kubernetes v1.28.6 default #10810, @mzaian)
• Make kubernetes v1.29.1 default
  Remove SecCompDefault feature gate from hardening configuration for kubernetes 1.29 ([kubernetes] Support kubernetes 1.29 #10820, @tmurakam)
• Make kubernetes v1.29.2 default ([kubernetes] Add hashes for kubernetes 1.29.2, 1.28.7, 1.27.11 #10919, @mzaian)
• Make kubernetes v1.29.3 default ([kubernetes] Add hashes for kubernetes 1.29.3, 1.28.8, 1.27.12 #11035, @mzaian)
• Make kubernetes v1.29.4 default ([kubernetes] Add hashes for kubernetes 1.29.4, 1.28.9, 1.27.13 #11108, @mzaian)
• Make kubernetes v1.29.5 default ([kubernetes] Add hashes for kubernetes 1.29.5, 1.28.10, 1.27.14 #11196, @mzaian)
• Metallb: added metallb_namespace variable to parameterize namespace ([metallb] Add variable for the metallb namespace #11136, @oik741)
• OpenStack Cloud Controller Manager upgrade to 1.28.2 (Bump openstack-cloud-controller-manager from 1.25.3 to 1.28.2 #11174, @tico88612)
• Opensuse deployment is now tested in CI. (Enable opensuse in CI #11159, @VannTen)
• Add selinux-ng repo in Amazon Linux to install container-selinux (Add selinux-ng repo in amazon linux to install container-selinux #11182, @yankay)
• Add CI Image for Ubuntu 24.04 (Add CI Image for Ubuntu 24.04 #11167, @yankay)
• Allows .vagrant folder location to be configured (Allow vagrant directory location to be configurable #10718, @kri5)
• Prevent nodelocaldns to be OOM-killed (Promote nodelocaldns daemonset to system-node-critical #11056, @sathieu)
• Support Node Feature Discovery (Support Node Feature Discovery #10861, @yankay)
• Support Ubuntu 24.04 (CI: add Ubuntu 24.04 support #11132, @tico88612)
• Support following k8s version selection pause image (Support following k8s version selection pause image #10756, @my-git9)
• The variable old_dns_domains (list) can be used for backward compatibility when changing dns_domain (Allow the DNS stack to be backward compatible with an old dns_domain #10630, @VannTen)
• Update external huawei cloud controller to 0.26.6 (Update external huawei cloud controller to 0.26.6 #10824, @dabeck)
• Update external huawei cloud controller to 0.26.8 (Update external huawei cloud controller to 0.26.8 #11172, @dabeck)
• Update kube-vip to v0.8.0 (Update kube-vip to v0.8.0 #11156, @jisnardo)
• Update metrics server to v0.7.0 ([feat] Update metrics server to v0.7.0 #10856, @mzaian)
• Updated ingress controller version to 1.9.6 (updated ingress controller version #10868, @kundan2707)
• User has a possibility to modify Service type with "ingress_nginx_service_type" property in addons. (feat/ingress-nginx service if hostNetwork set to false #10925, @chrxmvtik)
• [Terraform-openstack] Added possibility to build an octavia loadbalancer for the Kubernetes Api. ([Terraform-Openstack] Add optional Octavia loadbalancer for Master Nodes #10924, @jaszil)
• [containerd] added distributed tracing config variables for containerd (containerd_tracing_enabled,containerd_tracing_endpoint,containerd_tracing_protocol, containerd_tracing_sampling_ratio,containerd_tracing_service_name ); it is disabled by default. ([containerd/tracing]: add distributed tracing config flags #11103, @ugur99)
• [download] add capability to specify alternative download mirrors for files (implement download mirrors support #8474, @cristicalin)
• [etcd] Default version to 3.5.12 for k8s 1.27 , 1.28 , 1.29 ([etcd] make etcd 3.5.12 default #11036, @mzaian)
• Minimum ansible-core version is now 2.16.4 (Upgrade ansible-core to 2.16.4 #10984, @VannTen)
• Remove the archived debian apt repository when installing docker-engine (fix: Remove the archived debian apt repository when installing docker-engine #11088, @yankay)
• Change dependbot interval to weekly (Change dependbot interval to weekly #11189, @yankay)
• Allow specifying CPU Manager Policy options through kubelet_cpu_manager_policy_options (Allow specifying CPU Manager Policy Options #11023, @derselbst)
• [kube-apiserver] added distributed tracing config variables for kube-apiserver (kube_apiserver_tracing,kube_apiserver_tracing_endpoint,kube_apiserver_tracing_sampling_rate_per_million); it is disabled by default.
  [kubelet] added distributed tracing config variables for kubelet (kubelet_tracing,kubelet_tracing_endpoint,kubelet_tracing_sampling_rate_per_million); it is disabled by default. ([apiserver-kubelet/tracing]: add distributed tracing config variables #10795, @ugur99)

Applications

• [argocd] update argocd to v2.11.0 ([argocd] update argocd to v2.11.0 #11193, @mzaian)
• [helm] Upgrade to v3.14.2 (Upgrade Helm to v3.14.2 #10967, @cleman95)
• Bump coredns version to 1.11.1 (bump coredns version to 1.11.1 #10719, @batazor)
• Support CoreDNS use host network & config CoreDNS port (support CoreDNS use host network and config dns port #10617, @liuxu623)
• Fix secondary coredns missing var (Fix secondary coredns missing var #10821, @VannTen)
• dns_mode: coredns_dual is now tested in CI. (Add test for coredns_dual in CI #10903, @VannTen)

Network

• Adds support for cilium v1.15
  • Adds support for cilium_l2announcements to replace metallb with cilium l2 announcements, defaults to false
  • Adds support for cilium_loadbalancer_mode to switch bpf-lb-mode between snat, dsr or hybrid, default to snat (feat: Add support for cilium 1.15 and update cilium to v1.15.4 #11106, @deveshk0)
• Adds the option to install calico 3.27.3 (Add calico v3.27.3 checksums #11059, @danielfrg)
• [calico] Update default calico to v3.27.2 ([calico] Add hashes and make v3.27.2 default #10960, @mzaian)

Container-Managers

• crictl stop container grace period, cri_stop_containers_grace_period: 0 (crictl allow setting grace period for stop containers upon reset #10651, @krembu)
• Update the docker default version to 24.0 (Update Docker version to v24.0 #10873, @yankay)
• [Containerd] Enable by default discard_unpacked_layers to save some space (see Can I reduce the size of images in persistent storage (/var/lib/containerd)? containerd/containerd#6295) (Enable containerd 'discard_unpacked_layers' by default #10905, @VannTen)
• [Nerdctl] Upgrade to version 1.7.4 (Upgrade Nerdctl version to 1.7.4 #10968, @cleman95)
• [containerd] Make containerd 1.7.13 default
  [runc] Upgrade to v1.1.12 (make containerd 1.7.13 default and upgrade runc to v1.1.12 #10862, @KubeKyrie)
• [containerd] Make containerd 1.7.16 default ([containerd] add hashes for versions '1.6.30-31', 'v1.7.16' and make v1.7.16 default #11142, @mzaian)

API Change

• Make proxy protocol in Upcloud LB configurable (Make proxy protocol in upcloud LB configurable #10971, @davidumea)

Design

• Merge stop and remove systemd service task in reset/tasks/main.yml (Merge stops and removes systemd services tasks in reset playbook #10902, @kimsehwan96)

Documentation

• Add documentation for configuring nat outgoing ipv6 (Add documentation for configuring nat outgoing ipv6 #10866, @anders-elastisys)
• Add new OpenStack Cloud for terraform (Add new OpenStack Cloud for terraform #10910, @DragomirAlin)
• BREAKING CHANGE: This script is introduced to facilitate living documentation and its administration. This leads to a restructuring in the documentation at https://kubespray.io/#/ to simplify the automatic creation of links, as the structure in the sidebar changes. (add script to generate docs _sidebar.md based on the /docs content #11128, @Payback159)
• Change a task name Ensure kube-bench parameters are set into Ensure kubelet expected parameters are set in roles/kubernetes/preinstall/tasks/0080-system-configurations.yml for a clearer understanding of its operation (Change a task name in preinstall/0080-system-configurations.yml #11171, @kimsehwan96)
• Do not disable SELinux surreptitiously (do not disable SELinux surreptitiously #10920, @rptaylor)
• Doc clarification: skipping patches releases is OK (Doc clarification: skipping patches releases is OK #10850, @VannTen)
• Docs: vagrant-libvirt is tested in CI (docs: vagrant-libvirt is tested in CI #10847, @VannTen)
• Explicit private/public nature of *ip vars (Explicit private/public nature of *ip vars #10904, @VannTen)
• Fix typo in vagrant.md (Updated vagrant.md #10836, @kundan2707)
• Fix typo mistake in roles/kubernetes/control-plane/tasks/define-first-kube-control.yml (Fix typo mistake in roles/kubernetes/control-plane/tasks/define-first-kube-control.yml #10835, @kimsehwan96)
• Fixed typos in inventory/sample/group_vars/k8s_cluster (Fixed typos in inventory/sample/group_vars/k8s_cluster #10911, @arahmangulov)
• Kubespray used as a collection will have the correct collection version (Adjust the releases process documentation. #10727, @VannTen)
• Make large-deployments.md link to downloads.md (Make large-deployments.md link to downloads.md #10840, @spantaleev)
• Removed not needed graduated feature gates. (Fix Kubelet not starting because of non-existent feature gate #10448, @Smidra)
• Update upgrades.md with serial=1 for rolling updates (Update upgrades.md with serial=1 for rolling updates #10837, @titansmc)
• Variable cilium_ipsec_key must be base64 encoded (Doc: variable cilium_ipsec_key must be base64 encoded #10781, @ledroide)

Bug or Regression

• Added an optional variable (cni_bin_owner) to allow the user to set a different owner for /opt/cni/bin/ and it's contents. (Support overriding cni directory owner #10929, @Rickkwa)
• Change the position of the containerd_extra_args parameter to enhance its universality. (fix: change the position of the containerd_extra_args parameter to make the parameter more universal. #11013, @qcu266)
• Configure crio container runtime to use kube reserved cgroup (configure crio to use kube reserved cgroup #11028, @pedro-peter)
• Don't overwrite changes to openstack allowed_address_pairs don't overwrite changes to openstack allowed_address_pairs #10760 (don't overwrite changes to openstack allowed_address_pairs #10760, @rptaylor)
• Download cache directory permissions are no longer reset recursively (Don't try to set permissions recursively on cache+staging directory #10900, @VannTen)
• Fix ClusterRole for Calico >=v1.26.x with Calico API Server installed (fix: updating the calico-crds #11089, @RaSerge)
• Fix ansible parameter ssh_args in ansible.cfg file not work (fix ssh_args in ansible.cfg no effect #10981, @joy717)
• Fix boostrap for Amazon Linux (Fix amazon linux bootstrap #11139, @VannTen)
• Fix crio registries config file when using slashes in the registry path (crio container runtime crio_registries path fix #11030, @pedro-peter)
• Fix file loss during download (Fix file loss during download #10779, @ErikJiang)
• Fix kubespray-defaults: Check for boostrap-os FQCN (fix kubespray-defaults: Check for kubespray.boostrap-os FQCN #11073, @KubeKyrie)
• Fix local path provisioner image repo in sample inventory. (Fix: sample inventory local path provisioner image repo #11180, @tico88612)
• Fix logical error when checking for boostrap-os (Fix logical error when checking for boostrap-os #10867, @VannTen)
• Fix lsattr command error when kubelet has symbolic link (fix reset/main.yml lsattr command error when kubelet has symbolic link #11074, @KubeKyrie)
• Fix network manage service of Debian 12 (change debian12 network manage service from networking to systemd-netw… #11058, @KubeKyrie)
• Fix nginx controller leader election RBAC (Fix ingress-nginx controller election #10913, @VannTen)
• Fix python regex matching problem when finding docker packages (fix python regex matching problem when finding docker packages #11075, @KubeKyrie)
• Fix waiting for MetalLB controller (Fix waiting for MetalLB controller #10858, @flxbwr)
• Fix(kubernetes): taint nodes on cluster upgrade (fix(kubernetes): taint nodes on cluster upgrade #10705, @maxime1907)
• Fix: config hostname as string type in kubeadmConfig rendering (fix: config hostname as string type in kubeadmConfig rendering #10997, @ErikJiang)
• Fixes running recover-control-plane.yml with offline broken etcd nodes. (Offline control plane recover #10660, @yuha0)
• Revert OCCM standard dnsPolicy to ClusterFirst to fix OpenStack Cloud Provider Initialization Failure Due to DNSPolicy in DaemonSet Template #10914 which was introduced with add dnsPolicy: ClusterFirstWithHostNet to DaemonSets with hostNetwork: true #10618 and make dnsPolicy configurable to furthermore support add dnsPolicy: ClusterFirstWithHostNet to DaemonSets with hostNetwork: true #10618 (Revert OCCM standard dnsPolicy to ClusterFirst and make dnsPolicy con… #11168, @Payback159)
• Force update helm repo if exists on host (force update helm repo if exists on host #11043, @LuckySB)
• Kubespray ansible version checks are now performed even when running with --limit (Use run_once for checking ansible version instead of localhost #10908, @VannTen)
• None. (Update kubelet systemd service default allowed IP addresses for cluster hardening #11061, @bmelbourne)
• Revert crictl version (revert crictl version #11042, @ErikJiang)
• The script manage-offline-container-images.sh now supports additional environment variables, e.g. it is now possible use the script to pull images listed in a file instead of checking images in a running cluster. (Fix and improve manage-offline-container-images script #10857, @anders-elastisys)
• Update Snapshot Controller to 7.0.2 for all supported Kubernetes versions. (Update Volume Snapshot Controller to 7.0.2 #11041, @jess-sol)
• Workaround for terraform bug related to no_floating and extra_groups (workaround for terraform bug related to no_floating and extra_groups #10764, @rptaylor)
• [etcd] fixes wrong distributed tracing flag for etcd ([etcd/tracing]: fix etcd sampling rate flag #11175, @ugur99)
• Correct the POLY1305 cipher suites by adding the suffix _SHA256 (Enhance documentation for cipher suites #10641, @yckaolalala)

Other (Cleanup or Flake)

• Allow scripts/download_hash.sh to complete if new versions locations are changed of certain binaries (scripts: ignore download_hash download failures #10998, @MrFreezeex)
• Ddownload: Remove deleted kubeadm config field (download: Remove deleted kubeadm config field #10931, @VannTen)
• Download_file playbook - Incorrect description of a task (Download_file playbook - Incorrect description of a task #10875, @gianmarco-mameli)
• EventRecordQPS (in kubelet config) now uses Kubernetes default value (50) (Adjust kubelet_event_record_qps to K8S default #10826, @VannTen)
• Fixup galaxy.yml (Adjust galaxy.yml #10906, @VannTen)
• Ipaddr deprecation cleanup (using fqdn ansible.utils) (ipaddr deprecation cleanup #10822, @VannTen)
• Kubeadm images are now only downloaded where needed (Only download kubeadm images where needed #10899, @VannTen)
• Move perma failing jobs in CI to manual run (Move perma failing jobs in CI to manual run #10886, @VannTen)
• Remove dead link in README for debian jessie (Remove dead link in README for debian jessie #10827, @VannTen)
• Remove mirwan from approvers (Remove mirwan from approvers #10930, @VannTen)
• Update KUBESPRAY_VERSION for v2.24.0 (Update KUBESPRAY_VERSION in CI and Readme for v2.24.0 #10811, @yankay)
• Update KUBESPRAY_VERSION for v2.24.1 (Update KUBESPRAY_VERSION in CI and Readme for v2.24.1 #10962, @yankay)
• Update cinder-csi from 1.22.0 to 1.29.0
• Use the new container registry for cinder-csi (Bump cinder-csi version and switch container registry #10894, @Krast76)

Supported Components

• Core
  • kubernetes v1.29.5
  • etcd v3.5.12
  • docker v24.0 (see Note)
  • containerd v1.7.16
  • cri-o v1.29.1 (experimental: see CRI-O Note. Only on fedora, ubuntu and centos based OS)
• Network Plugin
  • cni-plugins v1.2.0
  • calico v3.27.3
  • cilium v1.15.4
  • flannel v0.22.0
  • kube-ovn v1.11.5
  • kube-router v2.0.0
  • multus v3.8
  • weave v2.8.1
  • kube-vip v0.8.0
• Application
  • cert-manager v1.13.2
  • coredns v1.11.1
  • ingress-nginx v1.10.1
  • krew v0.4.4
  • argocd v2.11.0
  • helm v3.14.2
  • metallb v0.13.9
  • registry v2.8.1
• Storage Plugin
  • cephfs-provisioner v2.1.0-k8s1.11
  • rbd-provisioner v2.1.1-k8s1.11
  • aws-ebs-csi-plugin v0.5.0
  • azure-csi-plugin v1.10.0
  • cinder-csi-plugin v1.29.0
  • gcp-pd-csi-plugin v1.9.2
  • local-path-provisioner v0.0.24
  • local-volume-provisioner v2.5.0
  • node-feature-discovery v0.14.2

Known issues

N/A

[sathieu]

Please could you fix #10860 before the release 🙏 ? A proposed fix is WIP in #10701 by @VannTen.

[poblahblahblah]

Please could you fix #10860 before the release 🙏 ? A proposed fix is WIP in #10701 by @VannTen.

I agree, this needs to get fixed before a new release is cut.

#10860 should arguably be included in a patch to the 2.24 branch as it's actively causing issues to production clusters.

[tico88612]

@mzaian Should we consider adding Support Ubuntu 24.04 #11132 to v2.25.0?

[sathieu]

Also #10988 (a feature) would be great!

[safetylab]

what about 1.30?

[spnngl]

Could we get #11123 and #11081 merged for this release please ? 🙏

[anders-elastisys]

A fix for #10914 would be nice. A potential fix could be to just add the necessary tolerations to coredns as seen here.

[mzaian]

This is a release draft and will be updated soon before we make a freeze.

[yankay]

This is a release draft and will be updated soon before we make a freeze.

HI @mzaian

You are welcome to ping me when it is ready to release.
I will create the GitHub tag/release for it.

[mzaian]

This is a release draft and will be updated soon before we make a freeze.

HI @mzaian

You are welcome to ping me when it is ready to release. I will create the GitHub tag/release for it.

@yankay Updated the draft release as well as the issue here.

/cc @floryut @VannTen @MrFreezeex

[mzaian]

@mzaian Should we consider adding Support Ubuntu 24.04 #11132 to v2.25.0?

I believe it's too early to work on supporting 24.04 so maybe in the next release.

[dibi-codes]

Also #11172 would be nice 😊

[tico88612]

@mzaian Hopefully #11174 will be adopted. (Upgrade OCCM to Kubespray's current K8s version)

[arusa]

Any plans when this is going to be released so we can upgrade to 1.29 (which was released 5 months ago)?

[mzaian]

Any plans when this is going to be released so we can upgrade to 1.29 (which was released 5 months ago)?

Hopefully next week, I will check all the pending issues/PRs on Monday and provide a status update.

[yankay]

Thanks @mzaian
Great Job.
/lgtm

[mhkarimi1383]

Any plans when this is going to be released so we can upgrade to 1.29 (which was released 5 months ago)?

Hopefully next week, I will check all the pending issues/PRs on Monday and provide a status update.

@yankay @mzaian

Hi
when you could get this update? it's been a week since last comment :)

[mzaian]

Any plans when this is going to be released so we can upgrade to 1.29 (which was released 5 months ago)?

Hopefully next week, I will check all the pending issues/PRs on Monday and provide a status update.

@yankay @mzaian

Hi when you could get this update? it's been a week since last comment :)

Please be patient We are awaiting some PRs to be finalised for the release.

[sathieu]

2.25.0 was released 2 days ago, thanks !

@mzaian FYI d8a4aea (#11185) is missing from release notes

[mzaian]

2.25.0 was released 2 days ago, thanks !

@mzaian FYI d8a4aea (#11185) is missing from release notes

Thanks I added it to the release notes.

[dibi-codes]

Will there be a bug fix release 1.25.1 anytime soon?