Operator for Azure Managed Service Identity in Kubernetes (for aad-pod-identity)

❗ IMPORTANT: Microsoft has deprecated aad-pod-identity service, please switch to azure-workload-identity

Operator for Azure Managed Service Identity (MSI) in Kubernetes, requires Azure aad-pod-identity service

Why using this app? Because it can be a security issue if developers can create AzureIdentity resources and could take over other teams Azure UserAssignedIdentity (MSI) resources.

This operator automates the process and detaches them from the developers responsibility. It looks up the configured namespaces (default configuration) and syncs AzureIdentity resources into the specified Kubernetes namespace. Then it checks AzureIdentityBinding resources for labels to bind AzureIdentity and AzureIdentityBinding together in a secure way.

Features

• automatically creates and maintains AzureIdentity resources in Kubernetes
• extracts Namespace from MSI tag resource (can be configured)
• automatically syncs AzureIdentity to AzureIdentityBinding using labels (simplifies deployments)
• allows to configure the name of AzureIdentity and namespace settings
• support expiry of AzureIdentity resources (use (hjacobs/kube-janitor)[https://codeberg.org/hjacobs/kube-janitor])
• leader election support (allows to run the operator multiple times with fast handover)
• supports Namespace creation and AzureIdentityBinding creating and modification watch in Kubernetes (allows fast and intelligent sync)
• exposes Prometheus metrics

Usage

Usage:
  azure-msi-operator [OPTIONS]

Application Options:
      --debug                                debug mode [$DEBUG]
  -v, --verbose                              verbose mode [$VERBOSE]
      --log.json                             Switch log output to json format [$LOG_JSON]
      --instance.nodename=                   Name of node where autopilot is running [$INSTANCE_NODENAME]
      --instance.namespace=                  Name of namespace where autopilot is running [$INSTANCE_NAMESPACE]
      --instance.pod=                        Name of pod where autopilot is running [$INSTANCE_POD]
      --lease.enable                         Enable lease (leader election; enabled by default in docker images) [$LEASE_ENABLE]
      --lease.name=                          Name of lease lock (default: azure-msi-operator-leader) [$LEASE_NAME]
      --sync.interval=                       Sync interval (time.duration) (default: 1h) [$SYNC_INTERVAL]
      --sync.watch                           Sync using namespace watch [$SYNC_WATCH]
      --sync.locktime=                       Lock time until next sync (time.duration) (default: 5m) [$SYNC_LOCKTIME]
      --azure.environment=                   Azure environment name (default: AZUREPUBLICCLOUD) [$AZURE_ENVIRONMENT]
      --azure.subscription=                  Azure subscription ID [$AZURE_SUBSCRIPTION_ID]
      --kubeconfig=                          Kuberentes config path (should be empty if in-cluster) [$KUBECONFIG]
      --kubernetes.label.format=             Kubernetes label format (sprintf, if empty, labels are not set) (default:
                                             msi.azure.k8s.io/%s) [$KUBERNETES_LABEL_FORMAT]
      --kubernetes.namespace.ignore=         Do not not maintain these namespaces (default: kube-system, kube-public, default,
                                             gatekeeper-system, istio-system) [$KUBERNETES_NAMESPACE_IGNORE]
      --azureidentity.namespaced             Set aadpodidentity.k8s.io/Behavior=namespaced annotation for AzureIdenity resources
                                             [$AZUREIDENTITY_NAMESPACED]
      --azureidentity.template.namespace=    Golang template for Kubernetes namespace (default: {{index .Tags "k8snamespace"}})
                                             [$AZUREIDENTITY_TEMPLATE_NAMESPACE]
      --azureidentity.template.resourcename= Golang template for Kubernetes resource name (default: {{ .Name }}-{{ .ClientId }})
                                             [$AZUREIDENTITY_TEMPLATE_RESOURCENAME]
      --azureidentity.binding.sync           Sync AzureIdentity to AzureIdentityBinding using lookup label
                                             [$AZUREIDENTITY_BINDING_SYNC]
      --azureidentity.expiry                 Enable setting of expiry for removal of old AzureIdentity resources (use with
                                             hjacobs/kube-janitor) [$AZUREIDENTITY_EXPIRY]
      --azureidentity.expiry.annotation=     Name of expiry annotation (default: janitor/expires)
                                             [$AZUREIDENTITY_EXPIRY_ANNOTATION]
      --azureidentity.expiry.duration=       Duration of expiry value (time.Duration) (default: 2190h)
                                             [$AZUREIDENTITY_EXPIRY_DURATION]
      --azureidentity.expiry.timeformat=     Format of absolute time (default: 2006-01-02) [$AZUREIDENTITY_EXPIRY_TIMEFORMAT]
      --server.bind=                         Server address (default: :8080) [$SERVER_BIND]
      --server.timeout.read=                 Server read timeout (default: 5s) [$SERVER_TIMEOUT_READ]
      --server.timeout.write=                Server write timeout (default: 10s) [$SERVER_TIMEOUT_WRITE]

Help Options:
  -h, --help                                 Show this help message

for Azure API authentication (using ENV vars) see https://docs.microsoft.com/en-us/azure/developer/go/azure-sdk-authentication

Example

Creates and maintains AzureIdentity resources in Kubernetes in an automated and safe way when found in Azure:

Example Azure MSI:

ResourceName: foobar
ResourceGroup: barfoo
Subscription: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
ClientID: df398181-f42f-41b4-b791-b1d4572be315
Tags:
    # separate multiple namespaces with comma
    k8snamespace: test123

Creates Kubernetes AzureIdentity:

apiVersion: "aadpodidentity.k8s.io/v1"
kind: AzureIdentity
metadata:
  name: foobar-df398181-f42f-41b4-b791-b1d4572be315
  namespace: test123
  labels:
    msi.azure.k8s.io/name: foobar
    msi.azure.k8s.io/resourcegroup: barfoo
    msi.azure.k8s.io/subscription: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  annotations:
      aadpodidentity.k8s.io/Behavior: namespaced #optional if namespaced mode is enabled
      janitor/expires: "2021-11-28" #optional if expiry is enabled
spec:
  type: 0
  resourceID: /subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourcegroups/barfoo/providers/Microsoft.ManagedIdentity/userAssignedIdentities/foobar
  clientID: df398181-f42f-41b4-b791-b1d4572be315

Syncs to AzureIdentityBinding (to allow recreation eg in development environments)

apiVersion: aadpodidentity.k8s.io/v1
kind: AzureIdentityBinding
metadata:
  labels:
    # used for sync AzureIdentity (eg. if recreated) to AzureIdentityBinding
    # if --azureidentitybinding.sync is used
    msi.azure.k8s.io/name: foobar
    msi.azure.k8s.io/resourcegroup: barfoo
    msi.azure.k8s.io/subscription: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  name: foobar
  namespace: test123
spec:
  azureIdentity: foobar-df398181-f42f-41b4-b791-b1d4572be315
  selector: your-selector

Templates

golang templates are used to offer flexible customization for namespace (--azureidentity.template.namespace) and resourcename (--azureidentity.template.resourcename) detection/creation, following information are available:

    Id               string
    Name             string
    Location         string
    ResourceGroup    string
    SubscriptionId   string
    ClientId         string
    TenantId         string
    PrincipalID      string
    Tags             map[string]string
    Type             string

Examples :

  env:
    # Use Azure ResourceName as AzureIdentity name (without ClientID)
    - name: AZUREIDENTITY_TEMPLATE_RESOURCENAME
      value "{{ .Name }}"

    # Use different Tag name for Namespace
    - name: AZUREIDENTITY_TEMPLATE_RESOURCENAME
      value: '{{index .Tags "namespace"}}'

Cleanup/expiry

This operator doesn't remove the AzureIdentity resources from your clusters to avoid any downtime because of eg. permissions issues in ServiceDiscovery. You can enable expiry annotations (AZUREIDENTITY_EXPIRY) and let them clean up with (hjacobs/kube-janitor)[https://codeberg.org/hjacobs/kube-janitor].

Metrics

Metric	Type	Description
azuremsi_sync_time	Gauge	Time (unix timestamp) of last sync run per Azure Subscription
azuremsi_sync_duration	Gauge	Duration of last sync per Azure Subscription
azuremsi_sync_resources_errors	Counter	Number of errors while syncing
azuremsi_sync_resources_success	Counter	Number of successfull syncs

AzureTracing metrics

(with 22.2.0 and later)

Azuretracing metrics collects latency and latency from azure-sdk-for-go and creates metrics and is controllable using environment variables (eg. setting buckets, disabling metrics or disable autoreset).

Metric	Description
azurerm_api_ratelimit	Azure ratelimit metrics (only on /metrics, resets after query due to limited validity)
azurerm_api_request_*	Azure request count and latency as histogram

Settings

Environment variable	Example	Description
METRIC_AZURERM_API_REQUEST_BUCKETS	1, 2.5, 5, 10, 30, 60, 90, 120	Sets buckets for azurerm_api_request histogram metric
METRIC_AZURERM_API_REQUEST_ENABLE	false	Enables/disables azurerm_api_request_* metric
METRIC_AZURERM_API_REQUEST_LABELS	apiEndpoint, method, statusCode	Controls labels of azurerm_api_request_* metric
METRIC_AZURERM_API_RATELIMIT_ENABLE	false	Enables/disables azurerm_api_ratelimit metric
METRIC_AZURERM_API_RATELIMIT_AUTORESET	false	Enables/disables azurerm_api_ratelimit autoreset after fetch

azurerm_api_request label	Status	Description
apiEndpoint	enabled by default	hostname of endpoint (max 3 parts)
routingRegion	enabled by default	detected region for API call, either routing region from Azure Management API or Azure resource location
subscriptionID	enabled by default	detected subscriptionID
tenantID	enabled by default	detected tenantID (extracted from jwt auth token)
resourceProvider	enabled by default	detected Azure Management API provider
method	enabled by default	HTTP method
statusCode	enabled by default	HTTP status code