k8tlery

Dissect container images, runtimes, and orchestrators.

Inventory

tool	scope	description
trivy		Find vulnerabilities, misconfigurations, secrets, SBOM in containers, Kubernetes, code repositories, clouds and more.
syft		CLI tool and library for generating a Software Bill of Materials from container images and filesystems.
grype		A vulnerability scanner for container images and filesystems.
kube-bench		Checks whether Kubernetes is deployed according to security best practices as defined in the CIS Kubernetes Benchmark.
checkov		Prevent cloud misconfigurations and find vulnerabilities during build-time in infrastructure as code, container images and open source packages with Checkov by Bridgecrew.
kubeaudit		kubeaudit helps you audit your Kubernetes clusters against common security controls.
cosign		Container Signing.
kdigger		Kubernetes focused container assessment and context discovery tool for penetration testing.
kubectl		Kubernetes provides a command line tool for communicating with a Kubernetes cluster's control plane, using the Kubernetes API.
docker		Command line interface for interacting with docker container images.
podman		A tool for managing OCI containers and pods.
dive		A tool for exploring a docker image, layer contents, and discovering ways to shrink the size of your Docker/OCI image.
crictl		CLI and validation tools for Kubelet Container Runtime Interface (CRI).
KubiScan		A tool to scan Kubernetes cluster for risky permissions.
Docker Bench Security		The Docker Bench for Security is a script that checks for dozens of common best-practices around deploying Docker containers in production.
peirates		Peirates, a Kubernetes penetration tool, enables an attacker to escalate privilege and pivot through a Kubernetes cluster. It automates known techniques to steal and collect service account tokens, secrets, obtain further code execution, and gain control of the cluster.
TruffleHog		Find and verify credentials.
TruffleHog3		This is an enhanced version of the Python-based truffleHog scanner.
Popeye		A Kubernetes cluster resource sanitizer.
k9s		Kubernetes CLI To Manage Your Clusters In Style.
Hadolint		Dockerfile linter, validate inline bash, written in Haskell.
Conftest		Write tests against structured configuration data using the Open Policy Agent Rego query language.
audit2rbac		Autogenerate RBAC policies based on Kubernetes audit logs.
kubeshark		The API traffic analyzer for Kubernetes providing real-time K8s protocol-level visibility, capturing and monitoring all traffic and payloads going in, out and across containers, pods, nodes and clusters. Inspired by Wireshark, purposely built for Kubernetes.
hardeneks		Runs checks to see if an EKS cluster follows EKS Best Practices.
amicontained		Container introspection tool. Find out what container runtime is being used as well as features available.
kubesec		Security risk analysis for Kubernetes resources.
kubectl-who-can		Show who has RBAC permissions to perform actions on different resources in Kubernetes.
etcdctl		etcdctl is a command line client for etcd.
gitleaks		Gitleaks is a SAST tool for detecting and preventing hardcoded secrets like passwords, api keys, and tokens in git repos. Gitleaks is an easy-to-use, all-in-one solution for detecting secrets, past or present, in your code.
kubeletctl		Kubeletctl is a command line tool that implement kubelet's API. Part of kubelet's API is documented but most of it is not. This tool covers all the documented and undocumented APIs.
kube-hunter		Hunt for security weaknesses in Kubernetes clusters.
netassert		Network security testing for Kubernetes DevSecOps workflows.
truffleproc		hunt secrets in process memory (TruffleHog & gdb mashup)
checkpointctl		Tool to inspect Kubernetes and Podman checkpoints.
...		...

Build & push

• build

docker buildx build -t ghcr.io/xopham/k8tlery:$K8TLERY_VERSION -t ghcr.io/xopham/k8tlery:latest .

• push

docker push ghcr.io/xopham/k8tlery --all-tags

• re-tag

find ./ -type f -exec sed -i "s%ghcr\.io\/xopham\/k8tlery\:v.*%ghcr\.io\/xopham\/k8tlery\:$K8TLERY_VERSION%g" {} \;

Usage

nix-shell

nix-shell k8tlery.nix

Docker

docker -it --rm ghcr.io/xopham/k8tlery:<tag>

Cluster

kubectl apply -f deployment/
#or
kubectl apply -f deployment/01-roles.yaml
kubectl apply -f deployment/02-k8tlery.yaml
#or
kubectl apply -f deployment/01-roles.yaml
kubectl apply -f deployment/03-k8tlery-fullaccess.yaml

kubectl exec -it k8tlery -- bash

Audit

Container image forensics

• download and save image

docker pull $IMAGE
docker save $IMAGE > image.tar
docker image ls

• inspect image content

docker inspect $IMAGE
docker history --no-trunc $IMAGE

• inspect image layers (dive)

dive $IMAGE

• extract file from image.tar (nix-shell custom functions)

layer_list $IMAGETAR $LAYERID $FILE  #run 'layer_list' for help
layer_extract $IMAGETAR $LAYERID $FILE  #run 'layer_list' for help

• create container w/o running it

docker create --name container $IMAGE  #returns container ID CONTID
docker container ls -a  #displays all available container IDs

• inspect container filesystems

mkdir $FOLDER
docker export $CONTID | tar -xC $FOLDER  #make sure to unpac to dedicated folder
ls -la $FOLDER

Container forensics

• create checkpoint of running container w/o interruption, e.g.:

sudo podman container checkpoint -e $OUTPUTFILE $CONTID --leave-running

• investigate checkpoint (checkpointctl)
  • get info

  checkpointctl show $OUTPUTFILE
  

  • get full details

  checkpointctl inspect $OUTPUTFILE --all
  

  • parse memory

  checkpointctl memparse #OUTPUTFILE --all

  • inspect container drift

  tar -xf $OUTPUTFILE -C $TARGETFOLDER
  tar -xf $TARGETFOLDER/rootfs-diff.tar $DIFFFOLDER

Cluster information gathering

• misconfiguration scan
  • trivy

  trivy k8s --report summary cluster

Pod/container information gathering

• container runtime

cat /proc/self/cgroup

• container runtime sockets (might be slow)

find /run -type f -name "*.sock"  #adjust target folder
# also need to review '/run' folder manually

• hosts information

cat /etc/hosts

• mount information

mount

• file system

ls -la /
ls -la /home/
ls -la /root/
ls -la /tmp/

• environment variables

printenv

• k8s information
  • kdigger

  curl -fSL -o /tmp/kdigger https://github.com/quarkslab/kdigger/releases/download/v1.5.0/kdigger-linux-amd64
  chmod +x /tmp/kdigger
  alias kdigger='/tmp/kdigger'
  kdigger dig all

  • kube-hunter

  pip3 install kube-hunter
  kube-hunter --pod

• secrets (trufflehog3)

pip3 install trufflehog3
trufflehog3 /var/run  #choose relevant target folders

* custom rule
```
#k8s-goat.rule
- id: k8s-goat.flag
  message: found k8s-goat flag
  pattern: "k8s-goat-"
  severity: HIGH
```
```bash
trufflehog3 -r k8s-goat.rule /tmp  #adjust rule and target
```

• secrets from process memory (truffleproc): needs work
• vulnerable packages (trivy)

curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
trivy rootfs /

• k8s APIs
  • curl

  APISERVER=https://${KUBERNETES_SERVICE_HOST}
  SERVICEACCOUNT=/var/run/secrets/kubernetes.io/serviceaccount
  NAMESPACE=$(cat ${SERVICEACCOUNT}/namespace)
  TOKEN=$(cat ${SERVICEACCOUNT}/token)
  CACERT=${SERVICEACCOUNT}/ca.crt
  curl --cacert ${CACERT} --header "Authorization: Bearer ${TOKEN}" -X GET ${APISERVER}/api

  • peirates

  curl -fSL -o /tmp/peirates.tar.xz https://github.com/inguardians/peirates/releases/download/v1.1.13/peirates-linux-amd64.tar.xz
  tar -xvf /tmp/peirates.tar.xz -C /tmp
  chmod a+x /tmp/peirates-linux-amd64/peirates
  alias peirates='/tmp/peirates-linux-amd64/peirates'
  peirates

Pod exploitation

• resource exhaustion (DoS)

stress-ng --cpu 2 --cpu-load 1 --vm 2 --vm-bytes 100m -t 100s --verify -v  #adjust to use case

• various angles (peirates)

curl -fSL -o /tmp/peirates.tar.xz https://github.com/inguardians/peirates/releases/download/v1.1.13/peirates-linux-amd64.tar.xz
tar -xvf /tmp/peirates.tar.xz -C /tmp
chmod a+x /tmp/peirates-linux-amd64/peirates
alias peirates='/tmp/peirates-linux-amd64/peirates'
peirates

References

• HackTricks (e.g. pentesting docker, pentesting kubernetes)
• Kubernetes Goat
• OWASP Kubernetes Security CS