Release Proposal v2.14

[floryut]

Below is a proposal for release notes:

Announcements

We are looking for maintainers, reach out in #5432.

Deprecation / Removal

• Removed support for Fedora 29 and 30 (EOL)
• Remove support for CoreOS Container Linux (EOL)

Major changes:

• Add Oracle Linux 8 support and fixes (Oracle Linux 8 support and fixes #6198)
• Add Ubuntu 20.04 support (Add Ubuntu 20.04 support #6157)
• Add support for Fedora 32 (Move fedora30 jobs to fedora32 #6426)
• Add support for Kata Containers (Add support for Kata Containers #6256)
• Switch to Python3 on Debian & Ubuntu (Add Ubuntu 20.04 support #6157)
• Add Ambassador OSS ingress controller (Support for Ambassador as an Ingress #6135)
• Add ovn4nfv-k8s-plugin as network plugin (adding ovn4nfv-k8s-plugin in kubespray #6381)
• Improve air-gap installation instructions (Improve air-gap installation instructions #6234)
• Add TLS cipher suites support for kubeadm and kubelet (ADD tls cipher suites support #6024 Add tlsMinVersion and tlsCipherSuites in kubelet configuration #6490)
• Update most ETCDCTL_API call to v3 (Make all etcd commands use ETCDCTL_API 3 #5998)
• Upgrade molecule to v3 (Upgrade Ansible molecule to version 3 #6468)
• Remove-node play will now fail if node can not be drained (make pre-remove node draining a failable task #6442)

Applications

• [Azure] Update documentation with az command (Update azure with az command #6042)
• [Azure] Add azure_cloud parameter to cloud_config file (added azure_cloud parameter to Azure's cloud_config #6321)
• [CSI] Update CSI containers to latest versions (Bump CSI containers to latest version #6221)
• [MetalLB] Option to talk BGP (Option for MetalLB to talk BGP #6383)
• [MetalLB] The deployment becomes one of addons. You can deploy it with a new option metallb_enabled (Change MetalLB to one of addons #6238)
• [Openstack] Support volume type (add master_volume_type TF variable to support volume type on Openstack #6524) (See Notes)
• [Openstack] Make it possible to open additional ports on masters ([Openstack] Make it possible to open additional ports on masters #6547)
• [Openstack] Add support for application credentials (Add support for openstack application credentials #6534)
• [Openstack] Add snapshot-controller for CSI drivers (Add snapshot-controller for CSI drivers and snapshot CRDs, add a default volumesnapshotclass when running cinder CSI #6537)
• [Openstack] Added a default volumesnapshotclass for Cinder CSI (Add snapshot-controller for CSI drivers and snapshot CRDs, add a default volumesnapshotclass when running cinder CSI #6537)

Container managers

• Match docker-cli version with docker-engine version (Fix docker-cli not compatible with docker-daemon version #6163)
• [Docker] Set cgroup driver by default to systemd (Docker: Set Cgroup driver by default to systemd #6563)
• [Containerd] Install package is now managed alongside docker (Manage containerd.io package with docker CRI. #6218)
• [Containerd] Add support for Fedora (Allow containerd runtime with fedora os (30/31) - add CI test #6094)
• [CRI-O] Use OS packaging default value for apparmor_profile in crio.conf (Fix #6118: Use OS packaging default for apparmor_profile in crio.conf #6125)
• [CRI-O] Fix kubelet cgroup driver detection (Fix kubelet cgroup driver detection for crio #6331)
• [CRI-O] Align template crio.conf with upstream and set cgroup driver by default to systemd (crio: align template crio.conf with upstream #6432)
• [CRI-O] Harden downloads with retry (crio: harden downloads with retry #6374)
• [CRI-O] Add variable to configure unsecure pull (cri-o: add variable to configure unsecure pull #6568)

Network

• [Weave] Allow Weave DS to support any taint effect (weave: support any taint effect in daemonset tolerations #6159)
• [Calico] Disable bird-check flag for probes of calico-node pods when calico_network_backend is not bird (disable bird-check flag when calico_network_backend is not 'bird' #6217)
• [Calico] Add FELIX_DEVICEROUTESOURCEADDRESS option (Deviceroutesourceaddress #6508)
• [Kube-Router] Enable portmap CNI plugin with kube-router to allow use of hostPort in container specs (Enable portmap CNI plugin with kube-router #6204)
• [Kube-Router] Add selectable dns policy (Add selectable dns policy for kube-router #6586)
• [Cilium] Add a way to deploy cilium alongside another CNI (Add a way to deploy cilium alongside another CNI #6373)
• [Cilium] Add option to configure IPVS timeouts in kube-proxy configration manifest (Add option to configure IPVS timeouts in kube-proxy configration #6396)
• [Cilium] Support the overwrite of MTU in Cilium agents (Cilium: overwrite auto-detected MTU of underlying network #6329)
• [Cilium] Add metrics in Cilium operator and add hubble metrics port in agents (Improve Cilium metrics support #6513)
• [Cilium] Add hubble server support in cilium (Add cilium hubble server in config #6575)

Other note worthy changes

• Create custom dashboard namespace if specified (Create namespace when dashboard deployment uses customized namespace. #6107)
• Add support to expose etcd metrics on a custom port (Add option to expose metrics on separate port #6092)
• Add additional network configuration options to external Openstack (Add additional network configuration options to external Openstack CCM (#6083) #6085)
• Fix resolv.conf configuration for Fedora CoreOS (Fix resolv.conf configuration for Fedora CoreOS. #6138)
• Replace seccomp profile docker/default with runtime/default (Replace seccomp profile docker/default with runtime/default #6170)
• Multiples fixes for proxy and no_proxy variables (Modify the populate no_proxy task to use a combine rather than relyin… #6112 Fix Issue 6430: add proxy_env definition to remove_node.yml #6431 Add proxy_env calculation to reset.yml #6558)
• Use connection: local when delegate_to: localhost (Use connection: local when delegate_to: localhost #6322)
• Add DNS configuration in NetworkManager for Fedora CoreOS (Use NetworkManager to manage resolv.conf for Fedora CoreOS #6291)
• Allow kubeadm to upgrade etcd (allow kubeadm to upgrade etcd #6345) (See notes)
• Add docs for setting up your first cluster (Add docs for 'setting up your first cluster' #6544)
• Webhook authorization can now be enabled using inventory variable (Allow webhook authorization #6502)
• Uncordon node that fail to drain (thus failing its upgrade) during upgrade procedure (Edited pre-upgrade task to uncordon a node failing to drain #6546)
• Added variable kubelet_rotate_server_certificates which enables kubelet server certificate rotation (Rotate kubelet server certificate. #6453)
• Add protectKernelDefaults option (default true) to kubelet config file (Add protectKernelDefaults option (default true) to kubelet config file #6611)

Component versions:

• Kubernetes v1.18.8
• Etcd 3.4.3
• Docker 19.03
• containerd 1.2.13
• Cri-O 1.18
• CNI-plugins v0.8.7
• Calico v3.15.2
• Cilium 1.8.3 (See Notes)
• Contiv 1.2.1
• Flannel 0.12.0
• Kube-Router 1.0.1 (see Notes)
• Multus 3.6
• kube-ovn 1.3.0 (see Notes)
• Weave 2.7.0
• CoreDNS 1.6.7
• nodelocaldns 1.15.13
• Helm 3.2.4
• nginx-ingress 0.35.0
• cert-manager 0.16.1 (see Notes)
• Kubernetes Dashboard v2.0.4
• Oracle OCI: v0.7.0

Known issues

TBD

Notes

• etcd will now be upgraded and its certs renewed when using a kubeadm managed etcd (etcd_kubeadm_enabled: true)
• Cilium: Check upgrade guide regarding update to 1.8.0
• Kube-Router: Upgrade to 1.0.0 require an iptable flush
• Kube-ovn is now installed in kube-system namespace, version priori to 1.0.0 should be removed manually
• Cert-Manager: Refer to README.md prior to upgrading in your exisitng Kubernetes cluster
• Openstack: If the nova API is before Stein, Terraform will work but the new volume type feature will not be available. If the entire cloud is upgraded to Stein or later, the new feature can be used. However if the nova versions in the cloud are mixed, with nova server API >= Stein and any nova-compute node < Stein, you will get a HTTP 409 error and VolumeTypeSupportNotYetAvailable exception.

[floryut]

@Miouge1 Let's save you some time when you got back ;)

[floryut]

Pending future PR merged @ 14/08/2020 / ef3e988

[lukasz-bielinski]

@floryut what is planned release date?

[floryut]

@floryut what is planned release date?

We haven't really discuss that yet (@EppO @LuckySB @mattymo ), but should be in the coming weeks.

[hafe]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

[floryut]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

[huxcrux]

I'm currently working on a PR that will fix snapshots for the Cinder CSI. There is some issue restoring a snapshot to volume when running the old v1alpha1 API. I hope to have the PR up in a few hours. If any other CSI drivers want the same fix I have in mind that everything need to be 100% reusable.

I hope I'm not to late and that we can have this feature added to the v2.14 release

[floryut]

I'm currently working on a PR that will fix snapshots for the Cinder CSI. There is some issue restoring a snapshot to volume when running the old v1alpha1 API. I hope to have the PR up in a few hours. If any other CSI drivers want the same fix I have in mind that everything need to be 100% reusable.

I hope I'm not to late and that we can have this feature added to the v2.14 release

Of course man, only wip release note, 2.14 is not till a few weeks 😃

[bmelbourne]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

FYI

I've raised PR #6563 to change the default cgroup driver to systemd for Docker.

Maybe we could consider getting this added to v2.14 to align with the CRI-O change?

[floryut]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

FYI

I've raised PR #6563 to change the default cgroup driver to systemd for Docker.

Maybe we could consider getting this added to v2.14 to align with the CRI-O change?

Let's wait for it to be finish (ie: not in WIP status)

[rptaylor]

I'm afraid I've run into an issue with #6524 on some clouds, it may be that the Openstack documentation is inaccurate in terms of the microversion requirement, or some other issue related to the use of the Openstack API, looking into it.

[Miouge1]

Thank you so much for this @floryut !

[rptaylor]

I'm pretty sure #6524 will be okay for 2.14.
The issue I ran into is described here: https://github.com/terraform-providers/terraform-provider-openstack/issues/1047
I think it is an Openstack deployment problem; if the cloud is only partially upgraded to >= Stein, the Terraform microversion check detects that the Openstack nova API can support the new feature and uses it, but the nova API checks the version of every compute node in the cloud, and if there is any older node it throws an exception. It should be fixed by fully upgrading all the nodes in the cloud to >= Stein.

[floryut]

I'm pretty sure #6524 will be okay for 2.14.
The issue I ran into is described here: terraform-providers/terraform-provider-openstack#1047
I think it is an Openstack deployment problem; if the cloud is only partially upgraded to >= Stein, the Terraform microversion check detects that the Openstack nova API can support the new feature and uses it, but the nova API checks the version of every compute node in the cloud, and if there is any older node it throws an exception. It should be fixed by fully upgrading all the nodes in the cloud to >= Stein.

Ok good to know :)

[floryut]

I do not know if this is worth a note:

• cgroup manager by default set to 'systemd'

Kind of backwards incompatible

Agreed could be added to network notes

FYI

I've raised PR #6563 to change the default cgroup driver to systemd for Docker.

Maybe we could consider getting this added to v2.14 to align with the CRI-O change?

And done ;)

[Miouge1]

Thank you so much @floryut ! v2.14.0 is out!