AAD integration support for Kuberenetes (#1049)

* Integrate AAD OpenID Connect support with Kubernetes in server side

* Update Kubernetes client kubeconfig for AAD support

* Add validation test for aadProfile

* Fix aad case

* Add document for AAD integration

Doc update

docs/README.md

@@ -12,6 +12,7 @@ This cluster definition examples demonstrate how to create a customized Docker E
 * [Kubernetes Walkthrough](kubernetes.md) - shows how to create a Kubernetes enabled Docker cluster on Azure
 * [Kubernetes Windows Walkthrough](kubernetes/windows.md) - shows how to create a hybrid Kubernetes Windows enabled Docker cluster on Azure.
 * [Kubernetes with GPU support Walkthrough](kubernetes.gpu.md) - shows how to create a Kubernetes cluster with GPU support.
+* [Kubernetes Kubernetes AAD integration Walkthrough](kubernetes.aad.md) - shows how to create a Kubernetes cluster with AAD as authentication provider.
 * [Swarm Walkthrough](swarm.md) - shows how to create a Swarm enabled Docker cluster on Azure
 * [Swarm Mode Walkthrough](swarmmode.md) - shows how to create a Swarm Mode cluster on Azure
 * [Custom VNET](../examples/vnet) - shows how to use a custom VNET

docs/clusterdefinition.md

@@ -156,3 +156,13 @@ For apiVersion "2016-03-30", a cluster may have only 1 agent pool profiles.
 |---|---|---|
 |adminUsername|yes|describes the username to be used on all linux clusters|
 |ssh.publicKeys[0].keyData|yes|The public SSH key used for authenticating access to all Linux nodes in the cluster.  Here are instructions for [generating a public/private key pair](ssh.md#ssh-key-generation).|
+
+### aadProfile
+
+`linuxProfile` provides [AAD integration](kubernetes.aad.md) configuration for the cluster, currently only available for Kubernetes orchestrator.
+
+|Name|Required|Description|
+|---|---|---|
+|clientAppID|yes|describes the client AAD application ID|
+|serverAppID|yes|describes the server AAD application ID|
+|tenantID|no|describes the AAD tenant ID to use for authentication. If not specified, will use the tenant of the deployment subscription.|

docs/kubernetes/aad.md

@@ -0,0 +1,108 @@
+# Microsoft Azure Container Service Engine - Kubernetes AAD integration Walkthrough
+
+This is walkthrough is to help you get start with Azure Active Directory(AAD) integeration with an ACS-Engine Kubernetes cluster.
+
+[OpenID Connect](http://openid.net/connect/) is a simple identity layer built on top of the OAuth 2.0 protocol, and it is supported by both AAD and Kubernetes. Here we're going to use OpenID Connect as the communication protocol.
+
+Please also refer to [Azure Active Directory plugin for client authentication](https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/plugin/pkg/client/auth/azure/README.md) in Kubernetes repo for more details abount OpenID Connect and AAD support in upstream.
+
+## Prerequision
+1. An Azure Active Directory tenant, will refer as `AAD Tenant`. You can use the tenant for your Azure subscription;
+2. A `Web app / API` type AAD application, will refer as `Server Application`. This application represents the `apiserver`;
+3. A `Native` type AAD application, will refer as `Client Application`. This application is for user login via `kubectl`. You'll need to add delegated permission to `Server Application`, please see [troubleshooting](#loginpageerror) section for detail.
+
+## Deployment
+Follow the [deployment steps](kubernetes.md#deployment). In step #4, add the following under 'properties' section:
+```
+"aadProfile": {
+    "serverAppID": "",
+    "clientAppID": "",
+    "tenantID": ""
+}
+```
+
+- `serverAppID`   : the `Server Application`'s ID
+- `clientAppID`   : the `Client Application`'s ID
+- `tenantID`      : (optional) the `AAD tenant`'s ID. If not specified, will use the tenant of the deployment subscription.
+
+After template generation, the local generated kubeconfig file (`_output/<instance>/kubeconfig/kubeconfig.<location>.json`) will have the default user using AAD.
+Initially it isn't assoicated with any AAD user yet. To get started, try any kubectl command (like `kubectl get pods`), and you'll be prompted to the device login process. After login, you will be able to operate the cluster using your AAD identity.
+
+### Note
+Please note that as of Kubernetes 1.7, the default is authorization mode is `AlwaysAllow`, which means any authenticated user have full access of the cluster.
+OpenID Connect is an authentication protocol responsible for identify users only, so initally all active accounts under the tenant will be able to login and have full admin privilege of the cluster.
+
+In this case you may want to also turn on RBAC for your cluster.
+Please refer to [Enable Kubernetes Role-Based Access Control](features.md#optional-enable-kubernetes-role-based-access-control-rbac) for turing on RBAC using acs-engine.
+
+Following instructions are for turnning on RBAC manually together with AAD integration:
+
+1. Since we use AAD object ID as OpenID Connect identity.
+    You'll first need to figure out your account's object ID. Here is how to do it using Azure Portal:
+    Navigate to `Azure Active Directory` -> `Users and groups` -> `All users`. And choose your account in right pannel. Switch to `Manage` -> `Profile`, and you can see the `Object ID` property.
+2. Figure out your user name. The user name would be in form of `IssuerUrl#ObjectID` format.
+    You can navigate to `https://login.microsoftonline.com/{tenantid}/.well-known/openid-configuration`, and find the `IssuerUrl` under `issuer` property.
+3. Add your account as admin role
+```
+kubectl create clusterrolebinding aad-default-cluster-admin-binding --clusterrole=cluster-admin --user={UserName}
+```
+For example, if your `IssuerUrl` is `https://sts.windows.net/e2917176-1632-47a0-ad18-671d485757a3/`, and your `ObjectID` is `22fa281b-bf62-4b14-972c-0dbca24a25a2`, the command would be:
+```
+kubectl create clusterrolebinding aad-default-cluster-admin-binding --clusterrole=cluster-admin --user=https://sts.windows.net/e2917176-1632-47a0-ad18-671d485757a3/#22fa281b-bf62-4b14-972c-0dbca24a25a2
+```
+
+4. Turn on RBAC on master nodes.
+    On master nodes, edit `/etc/kubernetes/manifests/kube-apiserver.yaml`, add `--authorization-mode=RBAC` under `command` property. Reboot nodes.
+5. Now that AAD account will be cluster admin, other accounts can still login but do not have permission for operating the cluster.
+    To verify this, add another client user:
+    ```
+    kubectl config set-credentials "user1" --auth-provider=azure \
+    --auth-provider-arg=environment=AzurePublicCloud \
+    --auth-provider-arg=client-id={ClientAppID} \
+    --auth-provider-arg=apiserver-id={ServerAppID} \
+    --auth-provider-arg=tenant-id={TenantID}
+    ```
+
+    And use that user to login
+    ```
+    kubectl get pods --user=user1
+    ```
+    Now you'll be prompted to login again, you can try logining with another AAD user account. 
+    The login would succeed, but later you can see following message since server denies access:
+    ```
+    Error from server (Forbidden): User "https://sts.windows.net/{tenantID}/#{objectID}" cannot list pods in the namespace "default". (get pods)
+    ```
+
+    You can manually update server configuration or add administrator users based on your requirement.
+
+## Troubleshooting
+
+### LoginPageError
+If you failed in login page, you may see following error message
+```
+Invalid resource. The client has requested access to a resource which is not listed in the requested permissions in the client's application registration. Client app ID: {UUID} Resource value from request: {UUID}. Resource app ID: {UUID}. List of valid resources from app registration: {UUID}.
+```
+This could be caused by `Client Application` not authorized.
+
+1. Go to Azure Portal, navigate to `Azure Active Directory` -> `App registrations`.
+2. Select the `Client Application`, Navigate to `Settings` -> `Required permissions`
+3. Choose `Add`, select the `Server Application`. In permissions tab, select `Delegated permissions` -> `Access {Server Application}`
+
+### ClientError
+If you see following message return from server via `kubectl`
+```
+Error from server (Forbidden)
+```
+
+It is usually caused by an incorrect configuration. You could find more debug information in apiserver log. On a master node, run following command:
+```
+docker logs -f $(docker ps|grep 'hyperkube apiserver'|cut -d' ' -f1) 2>&1 |grep -a auth
+```
+
+You might see following message like this:
+```
+Unable to authenticate the request due to an error: [invalid bearer token, [crypto/rsa: verification error, oidc: JWT claims invalid: invalid claims, 'aud' claim and 'client_id' do not match, aud=UUID1, client_id=spn:UUID2]]
+```
+This indicates server and client is using different `Server Application` ID, could usually happen when the configurations being updated manually.
+
+For other auth issues, you may also find some useful information from the log.

parts/kubeconfig.json

@@ -23,10 +23,7 @@
     "users": [
         {
             "name": "{{WrapAsVariable "resourceGroup"}}-admin",
-            "user": {
-                "client-certificate-data": "{{WrapAsVerbatim "variables('kubeConfigCertificate')"}}",
-                "client-key-data": "{{WrapAsVerbatim "variables('kubeConfigPrivateKey')"}}"
-            }
+            "user": {{authInfo}}
         }
     ]
 }

parts/kubernetesmaster-kube-apiserver.yaml

@@ -29,6 +29,9 @@ spec:
         - "--tls-private-key-file=/etc/kubernetes/certs/apiserver.key"
         - "--client-ca-file=/etc/kubernetes/certs/ca.crt"
         - "--service-account-key-file=/etc/kubernetes/certs/apiserver.key"
+        - "--oidc-client-id="
+        - "--oidc-issuer-url="
+        - "--oidc-username-claim=oid"
         - "--storage-backend=etcd2"
         - "--v=4"
         - "<kubernetesEnableRbac>"

parts/kubernetesmastercustomdata.yml

@@ -230,6 +230,22 @@ write_files:
     iptables -t nat -A POSTROUTING -m iprange ! --dst-range 168.63.129.16 -m addrtype ! --dst-type local ! -d {{WrapAsVariable "vnetCidr"}} -j MASQUERADE
 {{end}}
 
+{{ if .HasAadProfile }}
+    OIDC_CLIENT_ID=spn:{{WrapAsVariable "aadServerAppId"}}
+    VAR_AAD_TENANT_ID={{WrapAsVariable "aadTenantId"}}
+    VAR_TENANT_ID={{WrapAsVariable "tenantId"}}
+    VAR_TARGET_ENV={{WrapAsVariable "targetEnvironment"}}
+    AAD_TENANT_ID=${VAR_AAD_TENANT_ID:-$VAR_TENANT_ID}
+    AAD_ISSUER_HOST="sts.windows.net"
+    if [ "$VAR_TARGET_ENV" = "AzureChinaCloud" ]; then
+        AAD_ISSUER_HOST="sts.chinacloudapi.cn"
+    fi
+
+    OIDC_ISSUER_URL="https://$AAD_ISSUER_HOST/$AAD_TENANT_ID/" 
+    perl -pi -e "s|--oidc-client-id=\K(?=\")|$OIDC_CLIENT_ID| || s|--oidc-issuer-url=\K(?=\")|$OIDC_ISSUER_URL|" "/etc/kubernetes/manifests/kube-apiserver.yaml"
+{{else}}
+    sed -i "/--oidc-client-id\|--oidc-issuer-url\|--oidc-username-claim/d" "/etc/kubernetes/manifests/kube-apiserver.yaml"
+{{end}}
     sed -i "s|<kubernetesAddonManagerSpec>|{{WrapAsVariable "kubernetesAddonManagerSpec"}}|g" "/etc/kubernetes/manifests/kube-addon-manager.yaml"
     sed -i "s|<kubernetesHyperkubeSpec>|{{WrapAsVariable "kubernetesHyperkubeSpec"}}|g; s|<kubeServiceCidr>|{{WrapAsVariable "kubeServiceCidr"}}|g; s|<masterEtcdClientPort>|{{WrapAsVariable "masterEtcdClientPort"}}|g; s|<kubernetesAPIServerIP>|{{WrapAsVariable "kubernetesAPIServerIP"}}|g" "/etc/kubernetes/manifests/kube-apiserver.yaml"
     sed -i "s|<kubernetesHyperkubeSpec>|{{WrapAsVariable "kubernetesHyperkubeSpec"}}|g; s|<masterFqdnPrefix>|{{WrapAsVariable "masterFqdnPrefix"}}|g; s|<allocateNodeCidrs>|{{WrapAsVariable "allocateNodeCidrs"}}|g; s|<kubeClusterCidr>|{{WrapAsVariable "kubeClusterCidr"}}|g; s|<kubernetesCtrlMgrNodeMonitorGracePeriod>|{{WrapAsVariable "kubernetesCtrlMgrNodeMonitorGracePeriod"}}|g; s|<kubernetesCtrlMgrPodEvictionTimeout>|{{WrapAsVariable "kubernetesCtrlMgrPodEvictionTimeout"}}|g; s|<kubernetesCtrlMgrRouteReconciliationPeriod>|{{WrapAsVariable "kubernetesCtrlMgrRouteReconciliationPeriod"}}|g" "/etc/kubernetes/manifests/kube-controller-manager.yaml"

parts/kubernetesmastervars.t

@@ -50,6 +50,10 @@
     "masterVMSize": "[parameters('masterVMSize')]",
 {{end}}
     "sshPublicKeyData": "[parameters('sshRSAPublicKey')]",
+{{if .HasAadProfile}}
+    "aadServerAppId": "[parameters('aadServerAppId')]",
+    "aadTenantId": "[parameters('aadTenantId')]",
+{{end}}
 {{if not IsHostedMaster}}
   {{if GetClassicMode}}
     "masterCount": "[parameters('masterCount')]",

parts/kubernetesparams.t

@@ -1,3 +1,18 @@
+{{if .HasAadProfile}}
+    "aadServerAppId": {
+      "metadata": {
+        "description": "The server AAD application ID"
+      },
+      "type": "string"
+    },
+    "aadTenantId": {
+      "defaultValue": "",
+      "metadata": {
+        "description": "The AAD tenant ID to use for authentication. If not specified, will use the tenant of the deployment subscription."
+      },
+      "type": "string"
+    },
+{{end}}
     "apiServerCertificate": {
       "metadata": {
         "description": "The base 64 server certificate used on the master"
@@ -298,4 +313,3 @@
       },
       "type": "int"
     }
-

pkg/acsengine/engine.go

@@ -315,8 +315,25 @@ func GenerateKubeConfig(properties *api.Properties, location string) (string, er
 	kubeconfig = strings.Replace(kubeconfig, "{{WrapAsVerbatim \"variables('caCertificate')\"}}", base64.StdEncoding.EncodeToString([]byte(properties.CertificateProfile.CaCertificate)), -1)
 	kubeconfig = strings.Replace(kubeconfig, "{{WrapAsVerbatim \"reference(concat('Microsoft.Network/publicIPAddresses/', variables('masterPublicIPAddressName'))).dnsSettings.fqdn\"}}", FormatAzureProdFQDN(properties.MasterProfile.DNSPrefix, location), -1)
 	kubeconfig = strings.Replace(kubeconfig, "{{WrapAsVariable \"resourceGroup\"}}", properties.MasterProfile.DNSPrefix, -1)
-	kubeconfig = strings.Replace(kubeconfig, "{{WrapAsVerbatim \"variables('kubeConfigCertificate')\"}}", base64.StdEncoding.EncodeToString([]byte(properties.CertificateProfile.KubeConfigCertificate)), -1)
-	kubeconfig = strings.Replace(kubeconfig, "{{WrapAsVerbatim \"variables('kubeConfigPrivateKey')\"}}", base64.StdEncoding.EncodeToString([]byte(properties.CertificateProfile.KubeConfigPrivateKey)), -1)
+
+	var authInfo string
+	if properties.AADProfile == nil {
+		authInfo = fmt.Sprintf("{\"client-certificate-data\":\"%v\",\"client-key-data\":\"%v\"}",
+			base64.StdEncoding.EncodeToString([]byte(properties.CertificateProfile.KubeConfigCertificate)),
+			base64.StdEncoding.EncodeToString([]byte(properties.CertificateProfile.KubeConfigPrivateKey)))
+	} else {
+		tenantID := properties.AADProfile.TenantID
+		if len(tenantID) == 0 {
+			tenantID = "common"
+		}
+
+		authInfo = fmt.Sprintf("{\"auth-provider\":{\"name\":\"azure\",\"config\":{\"environment\":\"%v\",\"tenant-id\":\"%v\",\"apiserver-id\":\"%v\",\"client-id\":\"%v\"}}}",
+			GetCloudTargetEnv(location),
+			tenantID,
+			properties.AADProfile.ServerAppID,
+			properties.AADProfile.ClientAppID)
+	}
+	kubeconfig = strings.Replace(kubeconfig, "{{authInfo}}", authInfo, -1)
 
 	return kubeconfig, nil
 }
@@ -479,6 +496,11 @@ func getParameters(cs *api.ContainerService, isClassicMode bool) (paramsMap, err
 				addValue(parametersMap, "servicePrincipalClientSecret", properties.ServicePrincipalProfile.Secret)
 			}
 		}
+
+		if properties.AADProfile != nil {
+			addValue(parametersMap, "aadTenantId", properties.AADProfile.TenantID)
+			addValue(parametersMap, "aadServerAppId", properties.AADProfile.ServerAppID)
+		}
 	}
 
 	if strings.HasPrefix(properties.OrchestratorProfile.OrchestratorType, api.DCOS) {

pkg/api/converterfromapi.go

@@ -406,6 +406,10 @@ func convertPropertiesToVLabs(api *Properties, vlabsProps *vlabs.Properties) {
 		vlabsProps.CertificateProfile = &vlabs.CertificateProfile{}
 		convertCertificateProfileToVLabs(api.CertificateProfile, vlabsProps.CertificateProfile)
 	}
+	if api.AADProfile != nil {
+		vlabsProps.AADProfile = &vlabs.AADProfile{}
+		convertAADProfileToVLabs(api.AADProfile, vlabsProps.AADProfile)
+	}
 }
 
 func convertLinuxProfileToV20160930(api *LinuxProfile, obj *v20160930.LinuxProfile) {
@@ -811,3 +815,9 @@ func convertCertificateProfileToVLabs(api *CertificateProfile, vlabs *vlabs.Cert
 	vlabs.KubeConfigCertificate = api.KubeConfigCertificate
 	vlabs.KubeConfigPrivateKey = api.KubeConfigPrivateKey
 }
+
+func convertAADProfileToVLabs(api *AADProfile, vlabs *vlabs.AADProfile) {
+	vlabs.ClientAppID = api.ClientAppID
+	vlabs.ServerAppID = api.ServerAppID
+	vlabs.TenantID = api.TenantID
+}

pkg/api/convertertoapi.go

@@ -389,6 +389,11 @@ func convertVLabsProperties(vlabs *vlabs.Properties, api *Properties) {
 		api.CertificateProfile = &CertificateProfile{}
 		convertVLabsCertificateProfile(vlabs.CertificateProfile, api.CertificateProfile)
 	}
+
+	if vlabs.AADProfile != nil {
+		api.AADProfile = &AADProfile{}
+		convertVLabsAADProfile(vlabs.AADProfile, api.AADProfile)
+	}
 }
 
 func convertV20160930LinuxProfile(obj *v20160930.LinuxProfile, api *LinuxProfile) {
@@ -854,6 +859,12 @@ func convertVLabsCertificateProfile(vlabs *vlabs.CertificateProfile, api *Certif
 	api.KubeConfigPrivateKey = vlabs.KubeConfigPrivateKey
 }
 
+func convertVLabsAADProfile(vlabs *vlabs.AADProfile, api *AADProfile) {
+	api.ClientAppID = vlabs.ClientAppID
+	api.ServerAppID = vlabs.ServerAppID
+	api.TenantID = vlabs.TenantID
+}
+
 func addDCOSPublicAgentPool(api *Properties) {
 	publicPool := &AgentPoolProfile{}
 	// tag this agent pool with a known suffix string

pkg/api/types.go

@@ -51,6 +51,7 @@ type Properties struct {
 	JumpboxProfile          *JumpboxProfile          `json:"jumpboxProfile,omitempty"`
 	ServicePrincipalProfile *ServicePrincipalProfile `json:"servicePrincipalProfile,omitempty"`
 	CertificateProfile      *CertificateProfile      `json:"certificateProfile,omitempty"`
+	AADProfile              *AADProfile              `json:"aadProfile,omitempty"`
 	CustomProfile           *CustomProfile           `json:"customProfile,omitempty"`
 	HostedMasterProfile     *HostedMasterProfile     `json:"hostedMasterProfile,omitempty"`
 }
@@ -273,6 +274,18 @@ type HostedMasterProfile struct {
 	DNSPrefix string `json:"dnsPrefix"`
 }
 
+// AADProfile specifies attributes for AAD integration
+type AADProfile struct {
+	// The client AAD application ID.
+	ClientAppID string `json:"clientAppID,omitempty"`
+	// The server AAD application ID.
+	ServerAppID string `json:"serverAppID,omitempty"`
+	// The AAD tenant ID to use for authentication.
+	// If not specified, will use the tenant of the deployment subscription.
+	// Optional
+	TenantID string `json:"tenantID,omitempty"`
+}
+
 // CustomProfile specifies custom properties that are used for
 // cluster instantiation.  Should not be used by most users.
 type CustomProfile struct {
@@ -460,3 +473,8 @@ func (o *OrchestratorProfile) IsVNETIntegrated() bool {
 		return false
 	}
 }
+
+// HasAadProfile  returns true if the has aad profile
+func (p *Properties) HasAadProfile() bool {
+	return p.AADProfile != nil
+}

pkg/api/vlabs/types.go

@@ -38,6 +38,7 @@ type Properties struct {
 	WindowsProfile          *WindowsProfile          `json:"windowsProfile,omitempty"`
 	ServicePrincipalProfile *ServicePrincipalProfile `json:"servicePrincipalProfile,omitempty"`
 	CertificateProfile      *CertificateProfile      `json:"certificateProfile,omitempty"`
+	AADProfile              *AADProfile              `json:"aadProfile,omitempty"`
 }
 
 // ServicePrincipalProfile contains the client and secret used by the cluster for Azure Resource CRUD
@@ -245,6 +246,18 @@ type AgentPoolProfile struct {
 	CustomNodeLabels map[string]string `json:"customNodeLabels,omitempty"`
 }
 
+// AADProfile specifies attributes for AAD integration
+type AADProfile struct {
+	// The client AAD application ID.
+	ClientAppID string `json:"clientAppID,omitempty"`
+	// The server AAD application ID.
+	ServerAppID string `json:"serverAppID,omitempty"`
+	// The AAD tenant ID to use for authentication.
+	// If not specified, will use the tenant of the deployment subscription.
+	// Optional
+	TenantID string `json:"tenantID,omitempty"`
+}
+
 // KeyVaultSecrets specifies certificates to install on the pool
 // of machines from a given key vault
 // the key vault specified must have been granted read permissions to CRP