Skip to content

Latest commit

 

History

History
190 lines (146 loc) · 21.9 KB

plugin_agent_workloadattestor_k8s.md

File metadata and controls

190 lines (146 loc) · 21.9 KB
Raw

Agent plugin: WorkloadAttestor "k8s"

The k8s plugin generates Kubernetes-based selectors for workloads calling the agent. It does so by retrieving the workload's pod ID from its cgroup membership, then querying the kubelet for information about the pod.

The plugin can talk to the kubelet via the insecure read-only port or the secure port. Both X509 client authentication and bearer token (e.g. service account token) authentication to the secure port is supported.

Verifying the certificate presented by the kubelet over the secure port is optional. The default is to verify, based on the certificate file passed via kubelet_ca_path. skip_kubelet_verification can be set to disable verification.

The agent will contact the kubelet using the node name obtained via the node_name_env or node_name configurables. If a node name is not obtained, the kubelet is contacted over 127.0.0.1 (requires host networking to be enabled). In the latter case, the hostname is used to perform certificate server name validation against the kubelet certificate.

Note kubelet authentication via bearer token requires that the kubelet be started with the --authentication-token-webhook flag. See Kubelet authentication/authorization for details.

Note The kubelet uses the TokenReview API to validate bearer tokens. This requires reachability to the Kubernetes API server. Therefore API server downtime can interrupt workload attestation. The --authentication-token-webhook-cache-ttl kubelet flag controls how long the kubelet caches TokenReview responses and may help to mitigate this issue. A large cache ttl value is not recommended however, as that can impact permission revocation.

Note Anonymous authentication with the kubelet requires that the kubelet be started with the --anonymous-auth flag. It is discouraged to use anonymous auth mode in production as it requires authorizing anonymous users to the nodes/proxy resource that maps to some privileged operations, such as executing commands in containers and reading pod logs.

Note To run on Windows containers, Kubernetes v1.24+ and containerd v1.6+ are required, since hostprocess container is required on the agent container.

Configuration Description
disable_container_selectors If true, container selectors are not produced. This can be used to produce pod selectors when the workload pod is known but the workload container is not ready at the time of attestation.
kubelet_read_only_port The kubelet read-only port. This is mutually exclusive with kubelet_secure_port.
kubelet_secure_port The kubelet secure port. It defaults to 10250 unless kubelet_read_only_port is set.
kubelet_ca_path The path on disk to a file containing CA certificates used to verify the kubelet certificate. Required unless skip_kubelet_verification is set. Defaults to the cluster CA bundle /run/secrets/kubernetes.io/serviceaccount/ca.crt.
skip_kubelet_verification If true, kubelet certificate verification is skipped
token_path The path on disk to the bearer token used for kubelet authentication. Defaults to the service account token /run/secrets/kubernetes.io/serviceaccount/token
certificate_path The path on disk to client certificate used for kubelet authentication
private_key_path The path on disk to client key used for kubelet authentication
use_anonymous_authentication If true, use anonymous authentication for kubelet communication
node_name_env The environment variable used to obtain the node name. Defaults to MY_NODE_NAME.
node_name The name of the node. Overrides the value obtained by the environment variable specified by node_name_env.
experimental The experimental options that are subject to change or removal.
Experimental options Description
sigstore Sigstore options. Options described below. See Sigstore workload attestor for SPIRE
Sigstore options Description
skip_signature_verification_image_list The list of images, described as digest hashes, that should be skipped in signature verification. Defaults to empty list.
allowed_subjects_list A map of allowed subject strings, keyed by the OIDC Provider URI, that are trusted and are allowed to sign container images artifacts. Defaults to empty. If empty, no workload will pass signature validation, unless listed on skip_signature_verification_image_list. (eg. "https://accounts.google.com" = ["subject1@example.com","subject2@example.com"]).
rekor_url The rekor URL to use with cosign. Required. See notes below.
enforce_sct A boolean to be set to false in case of a private deployment, not using public CT

Note Cosign discourages the use of image tags for referencing docker images, and this plugin does not support attestation of sigstore selectors for workloads running on containers using tag-referenced images, which will then fail attestation for both sigstore and k8s selectors. In cases where this is necessary, add the digest string for the image in the skip_signature_verification_image_list setting (eg. "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789"). Note that sigstore signature attestation will still not be performed, but this will allow k8s selectors to be returned, along with the "k8s:sigstore-validation:passed" selector.

Note Since the SPIRE Agent can also go through workload attestation, it will also need to be included in the skip list if either its image is not signed or has a digest reference string.

Note The sigstore project contains a transparency log called Rekor that provides an immutable, tamper-resistant ledger to record signed metadata to an immutable record. While it is possible to run your own instance, a public instance of rekor is available at https://rekor.sigstore.dev/.

Sigstore workload attestor for SPIRE

Platform support

This capability is only supported on Unix systems.

The k8s workload attestor plugin also has capabilities to validate container images signatures through sigstore

Cosign supports container signing, verification, and storage in an OCI registry. Cosign aims to make signatures invisible infrastructure. For this, we’ve chosen the Sigstore ecosystem and artifacts. Digging deeper, we are using: Rekor (signature transparency log), Fulcio (signing certificate issuer and certificate transparency log) and Cosign (container image signing tool) to guarantee the authenticity of the running workload.

Note you can provide your own CA roots signed through TUF via the cosign initialize command. This effectively securely pins the CA roots. We allow you to also specify trusted roots via the SIGSTORE_ROOT_FILE flag

K8s selectors

Selector Value
k8s:ns The workload's namespace
k8s:sa The workload's service account
k8s:container-image The Image OR ImageID of the container in the workload's pod which is requesting an SVID, as reported by K8S. Selector value may be an image tag, such as: docker.io/envoyproxy/envoy-alpine:v1.16.0, or a resolved SHA256 image digest, such as docker.io/envoyproxy/envoy-alpine@sha256:bf862e5f5eca0a73e7e538224578c5cf867ce2be91b5eaed22afc153c00363eb
k8s:container-name The name of the workload's container
k8s:node-name The name of the workload's node
k8s:pod-label A label given to the workload's pod
k8s:pod-owner The name of the workload's pod owner
k8s:pod-owner-uid The UID of the workload's pod owner
k8s:pod-uid The UID of the workload's pod
k8s:pod-name The name of the workload's pod
k8s:pod-image An Image OR ImageID of any container in the workload's pod, as reported by K8S. Selector value may be an image tag, such as: docker.io/envoyproxy/envoy-alpine:v1.16.0, or a resolved SHA256 image digest, such as docker.io/envoyproxy/envoy-alpine@sha256:bf862e5f5eca0a73e7e538224578c5cf867ce2be91b5eaed22afc153c00363eb
k8s:pod-image-count The number of container images in workload's pod
k8s:pod-init-image An Image OR ImageID of any init container in the workload's pod, as reported by K8S. Selector value may be an image tag, such as: docker.io/envoyproxy/envoy-alpine:v1.16.0, or a resolved SHA256 image digest, such as docker.io/envoyproxy/envoy-alpine@sha256:bf862e5f5eca0a73e7e538224578c5cf867ce2be91b5eaed22afc153c00363eb
k8s:pod-init-image-count The number of init container images in workload's pod

Sigstore enabled selectors (available when configured to use sigstore)

Selector Value
k8s:${containerID}:image-signature-content A containerID is an unique alphanumeric number for each container. The value of the signature itself in a hash (eg. k8s:000000:image-signature-content:MEUCIQCyem8Gcr0sPFMP7fTXazCN57NcN5+MjxJw9Oo0x2eM+AIgdgBP96BO1Te/NdbjHbUeb0BUye6deRgVtQEv5No5smA=)
k8s:${containerID}:image-signature-subject OIDC principal that signed it​ (eg. k8s:000000:image-signature-subject:spirex@example.com)
k8s:${containerID}:image-signature-logid A unique LogID for the Rekor transparency log​ (eg. k8s:000000:image-signature-logid:samplelogID)
k8s:${containerID}:image-signature-integrated-time The time (in Unix timestamp format) when the image signature was integrated into the signature transparency log​ (eg. k8s:000000:image-signature-integrated-time:12345)
k8s:sigstore-validation The confirmation if the signature is valid, has value of "passed" (eg. k8s:sigstore-validation:passed)

Note container-image will ONLY match against the specific container in the pod that is contacting SPIRE on behalf of the pod, whereas pod-image and pod-init-image will match against ANY container or init container in the Pod, respectively.

Examples

To use the kubelet read-only port:

WorkloadAttestor "k8s" {
  plugin_data {
    kubelet_read_only_port = 10255
  }
}

To use the secure kubelet port, verify via /run/secrets/kubernetes.io/serviceaccount/ca.crt, and authenticate via the default service account token:

WorkloadAttestor "k8s" {
  plugin_data {
  }
}

To use the secure kubelet port, skip verification, and authenticate via the default service account token:

WorkloadAttestor "k8s" {
  plugin_data {
    skip_kubelet_verification = true
  }
}

To use the secure kubelet port, skip verification, and authenticate via some other token:

WorkloadAttestor "k8s" {
  plugin_data {
    skip_kubelet_verification = true
    token_path = "/path/to/token"
  }
}

To use the secure kubelet port, verify the kubelet certificate, and authenticate via an X509 client certificate:

WorkloadAttestor "k8s" {
  plugin_data {
    kubelet_ca_path = "/path/to/kubelet-ca.pem"
    certificate_path = "/path/to/cert.pem"
    private_key_path = "/path/to/key.pem"
  }
}

Platform support

This plugin is only supported on Unix systems.

Known issues

  • This plugin may fail to correctly attest workloads in pods that use lifecycle hooks to alter pod start behavior. This includes Istio workloads when the holdApplicationUntilProxyStarts configurable is set to true. Please see #3092 for more information. The disable_container_selectors configurable can be used to successfully attest workloads in this situation, albeit with reduced selector granularity (i.e. pod selectors only).