|
| 1 | +--- |
| 2 | +title: Configure Azure CNI Powered by Cilium in Azure Kubernetes Service (AKS) |
| 3 | +description: Learn how to create an Azure Kubernetes Service (AKS) cluster with Azure CNI Powered by Cilium. |
| 4 | +ms.topic: how-to |
| 5 | +ms.date: 02/12/2024 |
| 6 | +author: asudbring |
| 7 | +ms.author: allensu |
| 8 | +ms.subservice: aks-networking |
| 9 | +ms.custom: references_regions, devx-track-azurecli, build-2023, innovation-engine |
| 10 | +--- |
| 11 | + |
| 12 | +# Configure Azure CNI Powered by Cilium in Azure Kubernetes Service (AKS) |
| 13 | + |
| 14 | +Azure CNI Powered by Cilium combines the robust control plane of Azure CNI with the data plane of [Cilium](https://cilium.io/) to provide high-performance networking and security. |
| 15 | + |
| 16 | +By making use of eBPF programs loaded into the Linux kernel and a more efficient API object structure, Azure CNI Powered by Cilium provides the following benefits: |
| 17 | + |
| 18 | +- Functionality equivalent to existing Azure CNI and Azure CNI Overlay plugins |
| 19 | + |
| 20 | +- Improved Service routing |
| 21 | + |
| 22 | +- More efficient network policy enforcement |
| 23 | + |
| 24 | +- Better observability of cluster traffic |
| 25 | + |
| 26 | +- Support for larger clusters (more nodes, pods, and services) |
| 27 | + |
| 28 | +## IP Address Management (IPAM) with Azure CNI Powered by Cilium |
| 29 | + |
| 30 | +Azure CNI Powered by Cilium can be deployed using two different methods for assigning pod IPs: |
| 31 | + |
| 32 | +- Assign IP addresses from an overlay network (similar to Azure CNI Overlay mode) |
| 33 | + |
| 34 | +- Assign IP addresses from a virtual network (similar to existing Azure CNI with Dynamic Pod IP Assignment) |
| 35 | + |
| 36 | +If you aren't sure which option to select, read ["Choosing a network model to use."](./azure-cni-overlay.md#choosing-a-network-model-to-use) |
| 37 | + |
| 38 | +## Versions |
| 39 | + |
| 40 | +| Kubernetes Version | Cilium Version | |
| 41 | +|--------------------|----------------| |
| 42 | +| 1.27 (LTS) | 1.13.18 | |
| 43 | +| 1.28 (End of Life) | 1.13.18 | |
| 44 | +| 1.29 | 1.14.19 | |
| 45 | +| 1.30 (LTS) | 1.14.19 | |
| 46 | +| 1.31 | 1.16.6 | |
| 47 | +| 1.32 | 1.17.0 | |
| 48 | + |
| 49 | +See [Supported Kubernetes Versions](./supported-kubernetes-versions.md) for more information on AKS versioning and release timelines. |
| 50 | + |
| 51 | +## Network Policy Enforcement |
| 52 | + |
| 53 | +Cilium enforces [network policies to allow or deny traffic between pods](./operator-best-practices-network.md#control-traffic-flow-with-network-policies). With Cilium, you don't need to install a separate network policy engine such as Azure Network Policy Manager or Calico. |
| 54 | + |
| 55 | +## Limitations |
| 56 | + |
| 57 | +Azure CNI powered by Cilium currently has the following limitations: |
| 58 | + |
| 59 | +* Available only for Linux and not for Windows. |
| 60 | + |
| 61 | +* Cilium L7 policy enforcement is disabled. |
| 62 | + |
| 63 | +* Network policies can't use `ipBlock` to allow access to node or pod IPs. See [frequently asked questions](#frequently-asked-questions) for details and recommended workaround. |
| 64 | + |
| 65 | +* Multiple Kubernetes services can't use the same host port with different protocols (for example, TCP or UDP) ([Cilium issue #14287](https://github.com/cilium/cilium/issues/14287)). |
| 66 | + |
| 67 | +* Network policies may be enforced on reply packets when a pod connects to itself via service cluster IP ([Cilium issue #19406](https://github.com/cilium/cilium/issues/19406)). |
| 68 | + |
| 69 | +* Network policies aren't applied to pods using host networking (`spec.hostNetwork: true`) because these pods use the host identity instead of having individual identities. |
| 70 | + |
| 71 | +## Prerequisites |
| 72 | + |
| 73 | +* Azure CLI version 2.48.1 or later. Run `az --version` to see the currently installed version. If you need to install or upgrade, see [Install Azure CLI](/cli/azure/install-azure-cli). |
| 74 | + |
| 75 | +* If using ARM templates or the REST API, the AKS API version must be 2022-09-02-preview or later. |
| 76 | + |
| 77 | +> [!NOTE] |
| 78 | +> Previous AKS API versions (2022-09-02preview to 2023-01-02preview) used the field [`networkProfile.ebpfDataplane=cilium`](https://github.com/Azure/azure-rest-api-specs/blob/06dbe269f7d9c709cc225c92358b38c3c2b74d60/specification/containerservice/resource-manager/Microsoft.ContainerService/aks/preview/2022-09-02-preview/managedClusters.json#L6939-L6955). AKS API versions since 2023-02-02preview use the field [`networkProfile.networkDataplane=cilium`](https://github.com/Azure/azure-rest-api-specs/blob/06dbe269f7d9c709cc225c92358b38c3c2b74d60/specification/containerservice/resource-manager/Microsoft.ContainerService/aks/preview/2023-02-02-preview/managedClusters.json#L7152-L7173) to enable Azure CNI Powered by Cilium. |
| 79 | +
|
| 80 | +## Create a new AKS Cluster with Azure CNI Powered by Cilium |
| 81 | + |
| 82 | +### Create a Resource Group |
| 83 | + |
| 84 | +Use the following command to create a resource group. Environment variables are declared and used below to replace placeholders. |
| 85 | + |
| 86 | +```azurecli-interactive |
| 87 | +export RANDOM_SUFFIX=$(openssl rand -hex 3) |
| 88 | +export RESOURCE_GROUP="myResourceGroup$RANDOM_SUFFIX" |
| 89 | +export LOCATION="EastUS2" |
| 90 | +
|
| 91 | +az group create \ |
| 92 | + --name $RESOURCE_GROUP \ |
| 93 | + --location $LOCATION |
| 94 | +``` |
| 95 | + |
| 96 | +Result: |
| 97 | + |
| 98 | +<!-- expected_similarity=0.3 --> |
| 99 | +```JSON |
| 100 | +{ |
| 101 | + "id": "/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/myResourceGroupxxx", |
| 102 | + "location": "WestUS2", |
| 103 | + "name": "myResourceGroupxxx", |
| 104 | + "provisioningState": "Succeeded" |
| 105 | +} |
| 106 | +``` |
| 107 | + |
| 108 | +### Assign IP addresses from an overlay network |
| 109 | + |
| 110 | +Use the following commands to create a cluster with an overlay network and Cilium. Environment variables are declared and used below to replace placeholders. |
| 111 | + |
| 112 | +```azurecli-interactive |
| 113 | +export CLUSTER_NAME="myAKSCluster$RANDOM_SUFFIX" |
| 114 | +
|
| 115 | +az aks create \ |
| 116 | + --name $CLUSTER_NAME \ |
| 117 | + --resource-group $RESOURCE_GROUP \ |
| 118 | + --location $LOCATION \ |
| 119 | + --network-plugin azure \ |
| 120 | + --network-plugin-mode overlay \ |
| 121 | + --pod-cidr 192.168.0.0/16 \ |
| 122 | + --network-dataplane cilium \ |
| 123 | + --generate-ssh-keys |
| 124 | +``` |
| 125 | + |
| 126 | +<!-- expected_similarity=0.3 --> |
| 127 | +```JSON |
| 128 | +{ |
| 129 | + "id": "/subscriptions/xxxxx-xxxxx-xxxxx-xxxxx/resourceGroups/myResourceGroupxxx/providers/Microsoft.ContainerService/managedClusters/myAKSClusterxxx", |
| 130 | + "location": "WestUS2", |
| 131 | + "name": "myAKSClusterxxx", |
| 132 | + "provisioningState": "Succeeded" |
| 133 | +} |
| 134 | +``` |
| 135 | + |
| 136 | +> [!NOTE] |
| 137 | +> The `--network-dataplane cilium` flag replaces the deprecated `--enable-ebpf-dataplane` flag used in earlier versions of the aks-preview CLI extension. |
| 138 | +
|
| 139 | +## Frequently asked questions |
| 140 | + |
| 141 | +- **Can I customize Cilium configuration?** |
| 142 | + |
| 143 | + No, AKS manages the Cilium configuration and it can't be modified. We recommend that customers who require more control use [AKS BYO CNI](./use-byo-cni.md) and install Cilium manually. |
| 144 | + |
| 145 | +- **Can I use `CiliumNetworkPolicy` custom resources instead of Kubernetes `NetworkPolicy` resources?** |
| 146 | + |
| 147 | + `CiliumNetworkPolicy` custom resources are partially supported. Customers may use FQDN filtering as part of the [Advanced Container Networking Services](./advanced-container-networking-services-overview.md) feature bundle. |
| 148 | + |
| 149 | + This `CiliumNetworkPolicy` example demonstrates a sample matching pattern for services that match the specified label. |
| 150 | + |
| 151 | + ```yaml |
| 152 | + apiVersion: "cilium.io/v2" |
| 153 | + kind: CiliumNetworkPolicy |
| 154 | + metadata: |
| 155 | + name: "example-fqdn" |
| 156 | + spec: |
| 157 | + endpointSelector: |
| 158 | + matchLabels: |
| 159 | + foo: bar |
| 160 | + egress: |
| 161 | + - toFQDNs: |
| 162 | + - matchPattern: "*.example.com" |
| 163 | + ``` |
| 164 | +
|
| 165 | +- **Why is traffic being blocked when the `NetworkPolicy` has an `ipBlock` that allows the IP address?** |
| 166 | + |
| 167 | + A limitation of Azure CNI Powered by Cilium is that a `NetworkPolicy`'s `ipBlock` can't select pod or node IPs. |
| 168 | + |
| 169 | + For example, this `NetworkPolicy` has an `ipBlock` that allows all egress to `0.0.0.0/0`: |
| 170 | + ```yaml |
| 171 | + apiVersion: networking.k8s.io/v1 |
| 172 | + kind: NetworkPolicy |
| 173 | + metadata: |
| 174 | + name: example-ipblock |
| 175 | + spec: |
| 176 | + podSelector: {} |
| 177 | + policyTypes: |
| 178 | + - Egress |
| 179 | + egress: |
| 180 | + - to: |
| 181 | + - ipBlock: |
| 182 | + cidr: 0.0.0.0/0 # This will still block pod and node IPs. |
| 183 | + ``` |
| 184 | + |
| 185 | + However, when this `NetworkPolicy` is applied, Cilium blocks egress to pod and node IPs even though the IPs are within the `ipBlock` CIDR. |
| 186 | + |
| 187 | + As a workaround, you can add `namespaceSelector` and `podSelector` to select pods. This example selects all pods in all namespaces: |
| 188 | + ```yaml |
| 189 | + apiVersion: networking.k8s.io/v1 |
| 190 | + kind: NetworkPolicy |
| 191 | + metadata: |
| 192 | + name: example-ipblock |
| 193 | + spec: |
| 194 | + podSelector: {} |
| 195 | + policyTypes: |
| 196 | + - Egress |
| 197 | + egress: |
| 198 | + - to: |
| 199 | + - ipBlock: |
| 200 | + cidr: 0.0.0.0/0 |
| 201 | + - namespaceSelector: {} |
| 202 | + - podSelector: {} |
| 203 | + ``` |
| 204 | + |
| 205 | + > [!NOTE] |
| 206 | + > It isn't currently possible to specify a `NetworkPolicy` with an `ipBlock` to allow traffic to node IPs. |
| 207 | +- **Does AKS configure CPU or memory limits on the Cilium `daemonset`?** |
| 208 | + |
| 209 | + No, AKS doesn't configure CPU or memory limits on the Cilium `daemonset` because Cilium is a critical system component for pod networking and network policy enforcement. |
| 210 | + |
| 211 | +- **Does Azure CNI powered by Cilium use Kube-Proxy?** |
| 212 | + |
| 213 | + No, AKS clusters created with network dataplane as Cilium don't use Kube-Proxy. |
| 214 | + If the AKS clusters are on [Azure CNI Overlay](./azure-cni-overlay.md) or [Azure CNI with dynamic IP allocation](./configure-azure-cni-dynamic-ip-allocation.md) and are upgraded to AKS clusters running Azure CNI powered by Cilium, new nodes workloads are created without kube-proxy. Older workloads are also migrated to run without kube-proxy as a part of this upgrade process. |
| 215 | + |
| 216 | +## Next steps |
| 217 | + |
| 218 | +Learn more about networking in AKS in the following articles: |
| 219 | + |
| 220 | +* [Upgrade Azure CNI IPAM modes and Dataplane Technology](upgrade-azure-cni.md). |
| 221 | + |
| 222 | +* [Use a static IP address with the Azure Kubernetes Service (AKS) load balancer](static-ip.md) |
| 223 | + |
| 224 | +* [Use an internal load balancer with Azure Container Service (AKS)](internal-lb.md) |
| 225 | + |
| 226 | +* [Create a basic ingress controller with external network connectivity][aks-ingress-basic] |
| 227 | + |
| 228 | +<!-- LINKS - Internal --> |
| 229 | +[aks-ingress-basic]: ingress-basic.md |
0 commit comments