Understanding #Trivy Scan Discrepancies in EKS Pods

[chen-keinan]

Discussed in #6470

^{Originally posted by avaniicf April 8, 2024}

Description

Hi #trivy, we’ve been using Trivy to scan our EKS pods that deploy our application containers. Upon running a summary report against the pods, it’s indicating a HIGH severity with 4 failed checks. To delve into the specifics of these HIGH findings, we’ve conducted a detailed report on the same pods. However, we’ve noticed that the detailed report doesn’t explicitly list any HIGH findings. Upon closer examination, we discovered that certain vulnerabilities categorized as LOW in the detailed report are contributing to the overall HIGH severity indicated in the summary.

Desired Behavior

We expect to see a detailed report that resonates with the summary report provided by trivy scans.
If Summary shows a HIGH finding with 4 failed checks, we would expect the detailed report to point us to those failed checks under HIGH severity.

Actual Behavior

We’ve noticed that the detailed report doesn’t explicitly list any HIGH findings. In fact, certain vulnerabilities categorized as LOW in the detailed report are contributing to the overall HIGH severity indicated in the summary.

Reproduction Steps

1. Have an EKS cluster and pods to run the trivy scan against.
2. Run - `trivy k8s -n prd --compliance=k8s-cis --report summary deployment/xxxx-pod`
3. The `report summary` Output looks like this - (a section of how the summary report looks like)

Summary Report for compliance: CIS Kubernetes Benchmarks v1.23
┌────────┬──────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────┬────────┐
│   ID   │ Severity │                                                  Control Name                                                   │ Status │ Issues │
├────────┼──────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┼────────┤
│ 5.2.12 │  MEDIUM  │ Minimize the admission of HostPath volumes                                                                      │  PASS  │   0    │
│ 5.2.13 │  MEDIUM  │ Minimize the admission of containers which use HostPorts                                                        │  PASS  │   0    │
│ 5.3.1  │  MEDIUM  │ Ensure that the CNI in use supports Network Policies (Manual)                                                   │   -    │   -    │
│ 5.3.2  │  MEDIUM  │ Ensure that all Namespaces have Network Policies defined                                                        │  PASS  │   0    │
│ 5.4.1  │  MEDIUM  │ Prefer using secrets as files over secrets as environment variables (Manual)                                    │   -    │   -    │
│ 5.4.2  │  MEDIUM  │ Consider external secret storage (Manual)                                                                       │   -    │   -    │
│ 5.5.1  │  MEDIUM  │ Configure Image Provenance using ImagePolicyWebhook admission controller (Manual)                               │   -    │   -    │
│ 5.7.1  │  MEDIUM  │ Create administrative boundaries between resources using namespaces (Manual)                                    │   -    │   -    │
│ 5.7.2  │  MEDIUM  │ Ensure that the seccomp profile is set to docker/default in your pod definitions                                │  FAIL  │   1    │
│ 5.7.3  │   HIGH   │ Apply Security Context to Your Pods and Containers                                                              │  FAIL  │   4    │
│ 5.7.4  │  MEDIUM  │ The default namespace should not be used                                                                        │  PASS  │   0    │
└────────┴──────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────┴────────┘

The report all Output does not show any HIGH findings but LOWs and MEDIUMs.

 ~ trivy k8s -n prd --compliance=k8s-cis --report all deployment/xxxx-pod --severity HIGH
1 / 1 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 0 p/s

### Target

Kubernetes

### Scanner

Vulnerability

### Output Format

Table

### Mode

Standalone

### Debug Output

```bash
➜  ~ trivy k8s -n prd --compliance=k8s-cis --report summary deployment/xxxx-pod --debug
2024-04-08T10:32:54.676-0400	DEBUG	Severities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]
2024-04-08T10:32:54.676-0400	DEBUG	Ignore statuses	{"statuses": null}
2024-04-08T10:32:56.425-0400	DEBUG	cache dir:  /Users/amehta/Library/Caches/trivy
2024-04-08T10:32:56.426-0400	DEBUG	DB update was skipped because the local DB is the latest
2024-04-08T10:32:56.426-0400	DEBUG	DB Schema: 2, UpdatedAt: 2024-04-08 12:12:42.137892703 +0000 UTC, NextUpdate: 2024-04-08 18:12:42.137892442 +0000 UTC, DownloadedAt: 2024-04-08 14:32:27.417835 +0000 UTC
1 / 1 [-------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 0 p/s

Summary Report for compliance: CIS Kubernetes Benchmarks v1.23
┌────────┬──────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────┬────────┐
│   ID   │ Severity │                                                  Control Name                                                   │ Status │ Issues │
├────────┼──────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────┼────────┤
│ 1.1.1  │   HIGH   │ Ensure that the API server pod specification file permissions are set to 600 or more restrictive                │  PASS  │   0    │
│ 1.1.2  │   HIGH   │ Ensure that the API server pod specification file ownership is set to root:root                                 │  PASS  │   0    │
│ 1.1.3  │   HIGH   │ Ensure that the controller manager pod specification file permissions are set to 600 or more restrictive        │  PASS  │   0    │
│ 1.1.4  │   HIGH   │ Ensure that the controller manager pod specification file ownership is set to root:root                         │  PASS  │   0    │
│ 1.1.5  │   HIGH   │ Ensure that the scheduler pod specification file permissions are set to 600 or more restrictive                 │  PASS  │   0    │
│ 1.1.6  │   HIGH   │ Ensure that the scheduler pod specification file ownership is set to root:root                                  │  PASS  │   0    │
│ 1.1.7  │   HIGH   │ Ensure that the etcd pod specification file permissions are set to 600 or more restrictive                      │  PASS  │   0    │
│ 1.1.8  │   HIGH   │ Ensure that the etcd pod specification file ownership is set to root:root                                       │  PASS  │   0    │
│ 1.1.9  │   HIGH   │ Ensure that the Container Network Interface file permissions are set to 600 or more restrictive                 │  PASS  │   0    │
│ 1.1.10 │   HIGH   │ Ensure that the Container Network Interface file ownership is set to root:root                                  │  PASS  │   0    │
│ 1.1.11 │   HIGH   │ Ensure that the etcd data directory permissions are set to 700 or more restrictive                              │  PASS  │   0    │
│ 1.1.12 │   LOW    │ Ensure that the etcd data directory ownership is set to etcd:etcd                                               │  PASS  │   0    │
│ 1.1.13 │ CRITICAL │ Ensure that the admin.conf file permissions are set to 600                                                      │  PASS  │   0    │
│ 1.1.14 │ CRITICAL │ Ensure that the admin.conf file ownership is set to root:root                                                   │  PASS  │   0    │
│ 1.1.15 │   HIGH   │ Ensure that the scheduler.conf file permissions are set to 600 or more restrictive                              │  PASS  │   0    │
│ 1.1.16 │   HIGH   │ Ensure that the scheduler.conf file ownership is set to root:root                                               │  PASS  │   0    │
│ 1.1.17 │   HIGH   │ Ensure that the controller-manager.conf file permissions are set to 600 or more restrictive                     │  PASS  │   0    │
│ 1.1.18 │   HIGH   │ Ensure that the controller-manager.conf file ownership is set to root:root                                      │  PASS  │   0    │
│ 1.1.19 │ CRITICAL │ Ensure that the Kubernetes PKI directory and file ownership is set to root:root                                 │  PASS  │   0    │
│ 1.1.20 │ CRITICAL │ Ensure that the Kubernetes PKI certificate file permissions are set to 600 or more restrictive                  │  PASS  │   0    │
│ 1.1.21 │ CRITICAL │ Ensure that the Kubernetes PKI key file permissions are set to 600                                              │  PASS  │   0    │
│ 1.2.1  │  MEDIUM  │ Ensure that the --anonymous-auth argument is set to false                                                       │  PASS  │   0    │
│ 1.2.2  │   LOW    │ Ensure that the --token-auth-file parameter is not set                                                          │  PASS  │   0    │
│ 1.2.3  │   LOW    │ Ensure that the --DenyServiceExternalIPs is not set                                                             │  PASS  │   0    │
│ 1.2.4  │   LOW    │ Ensure that the --kubelet-https argument is set to true                                                         │  PASS  │   0    │
│ 1.2.5  │   HIGH   │ Ensure that the --kubelet-client-certificate and --kubelet-client-key arguments are set as appropriate          │  PASS  │   0    │
│ 1.2.6  │   HIGH   │ Ensure that the --kubelet-certificate-authority argument is set as appropriate                                  │  PASS  │   0    │
│ 1.2.7  │   LOW    │ Ensure that the --authorization-mode argument is not set to AlwaysAllow                                         │  PASS  │   0    │
│ 1.2.8  │   HIGH   │ Ensure that the --authorization-mode argument includes Node                                                     │  PASS  │   0    │
│ 1.2.9  │   HIGH   │ Ensure that the --authorization-mode argument includes RBAC                                                     │  PASS  │   0    │
│ 1.2.10 │   HIGH   │ Ensure that the admission control plugin EventRateLimit is set                                                  │  PASS  │   0    │
│ 1.2.11 │   LOW    │ Ensure that the admission control plugin AlwaysAdmit is not set                                                 │  PASS  │   0    │
│ 1.2.12 │  MEDIUM  │ Ensure that the admission control plugin AlwaysPullImages is set                                                │  PASS  │   0    │
│ 1.2.13 │  MEDIUM  │ Ensure that the admission control plugin SecurityContextDeny is set if PodSecurityPolicy is not used            │  PASS  │   0    │
│ 1.2.14 │   LOW    │ Ensure that the admission control plugin ServiceAccount is set                                                  │  PASS  │   0    │
│ 1.2.15 │   LOW    │ Ensure that the admission control plugin NamespaceLifecycle is set                                              │  PASS  │   0    │
│ 1.2.16 │   LOW    │ Ensure that the admission control plugin NodeRestriction is set                                                 │  PASS  │   0    │
│ 1.2.17 │   HIGH   │ Ensure that the --secure-port argument is not set to 0                                                          │  PASS  │   0    │
│ 1.2.18 │   LOW    │ Ensure that the --profiling argument is set to false                                                            │  PASS  │   0    │
│ 1.2.19 │   LOW    │ Ensure that the --audit-log-path argument is set                                                                │  PASS  │   0    │
│ 1.2.20 │   LOW    │ Ensure that the --audit-log-maxage argument is set to 30 or as appropriate                                      │  PASS  │   0    │
│ 1.2.21 │   LOW    │ Ensure that the --audit-log-maxbackup argument is set to 10 or as appropriate                                   │  PASS  │   0    │
│ 1.2.22 │   LOW    │ Ensure that the --audit-log-maxsize argument is set to 100 or as appropriate                                    │  PASS  │   0    │
│ 1.2.24 │   LOW    │ Ensure that the --service-account-lookup argument is set to true                                                │  PASS  │   0    │
│ 1.2.25 │   LOW    │ Ensure that the --service-account-key-file argument is set as appropriate                                       │  PASS  │   0    │
│ 1.2.26 │   LOW    │ Ensure that the --etcd-certfile and --etcd-keyfile arguments are set as appropriate                             │  PASS  │   0    │
│ 1.2.27 │  MEDIUM  │ Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate                     │  PASS  │   0    │
│ 1.2.28 │   LOW    │ Ensure that the --client-ca-file argument is set appropriate                                                    │  PASS  │   0    │
│ 1.2.29 │   LOW    │ Ensure that the --etcd-cafile argument is set as appropriate                                                    │  PASS  │   0    │
│ 1.2.30 │   LOW    │ Ensure that the --encryption-provider-config argument is set as appropriate                                     │  PASS  │   0    │
│ 1.3.1  │  MEDIUM  │ Ensure that the --terminated-pod-gc-threshold argument is set as appropriate                                    │  PASS  │   0    │
│ 1.3.3  │  MEDIUM  │ Ensure that the --use-service-account-credentials argument is set to true                                       │  PASS  │   0    │
│ 1.3.4  │  MEDIUM  │ Ensure that the --service-account-private-key-file argument is set as appropriate                               │  PASS  │   0    │
│ 1.3.5  │  MEDIUM  │ Ensure that the --root-ca-file argument is set as appropriate                                                   │  PASS  │   0    │
│ 1.3.6  │  MEDIUM  │ Ensure that the RotateKubeletServerCertificate argument is set to true                                          │  PASS  │   0    │
│ 1.3.7  │   LOW    │ Ensure that the --bind-address argument is set to 127.0.0.1                                                     │  PASS  │   0    │
│ 1.4.1  │  MEDIUM  │ Ensure that the --profiling argument is set to false                                                            │  PASS  │   0    │
│ 1.4.2  │ CRITICAL │ Ensure that the --bind-address argument is set to 127.0.0.1                                                     │  PASS  │   0    │
│ 2.1    │  MEDIUM  │ Ensure that the --cert-file and --key-file arguments are set as appropriate                                     │  PASS  │   0    │
│ 2.2    │ CRITICAL │ Ensure that the --client-cert-auth argument is set to true                                                      │  PASS  │   0    │
│ 2.3    │ CRITICAL │ Ensure that the --auto-tls argument is not set to true                                                          │  PASS  │   0    │
│ 2.4    │ CRITICAL │ Ensure that the --peer-cert-file and --peer-key-file arguments are set as appropriate                           │  PASS  │   0    │
│ 2.5    │ CRITICAL │ Ensure that the --peer-client-cert-auth argument is set to true                                                 │  PASS  │   0    │
│ 2.6    │   HIGH   │ Ensure that the --peer-auto-tls argument is not set to true                                                     │  PASS  │   0    │
│ 3.1.1  │   HIGH   │ Client certificate authentication should not be used for users (Manual)                                         │   -    │   -    │
│ 3.2.1  │   HIGH   │ Ensure that a minimal audit policy is created (Manual)                                                          │   -    │   -    │
│ 3.2.2  │   HIGH   │ Ensure that the audit policy covers key security concerns (Manual)                                              │   -    │   -    │
│ 4.1.1  │   HIGH   │ Ensure that the kubelet service file permissions are set to 600 or more restrictive                             │  PASS  │   0    │
│ 4.1.2  │   HIGH   │ Ensure that the kubelet service file ownership is set to root:root                                              │  PASS  │   0    │
│ 4.1.3  │   HIGH   │ If proxy kubeconfig file exists ensure permissions are set to 600 or more restrictive                           │  PASS  │   0    │
│ 4.1.4  │   HIGH   │ If proxy kubeconfig file exists ensure ownership is set to root:root                                            │  PASS  │   0    │
│ 4.1.5  │   HIGH   │ Ensure that the --kubeconfig kubelet.conf file permissions are set to 600 or more restrictive                   │  PASS  │   0    │
│ 4.1.6  │   HIGH   │ Ensure that the --kubeconfig kubelet.conf file ownership is set to root:root                                    │  PASS  │   0    │
│ 4.1.7  │ CRITICAL │ Ensure that the certificate authorities file permissions are set to 600 or more restrictive                     │  PASS  │   0    │
│ 4.1.8  │ CRITICAL │ Ensure that the client certificate authorities file ownership is set to root:root                               │  PASS  │   0    │
│ 4.1.9  │   HIGH   │ If the kubelet config.yaml configuration file is being used validate permissions set to 600 or more restrictive │  PASS  │   0    │
│ 4.1.10 │   HIGH   │ If the kubelet config.yaml configuration file is being used validate file ownership is set to root:root         │  PASS  │   0    │
│ 4.2.1  │ CRITICAL │ Ensure that the --anonymous-auth argument is set to false                                                       │  PASS  │   0    │
│ 4.2.2  │ CRITICAL │ Ensure that the --authorization-mode argument is not set to AlwaysAllow                                         │  PASS  │   0    │
│ 4.2.3  │ CRITICAL │ Ensure that the --client-ca-file argument is set as appropriate                                                 │  PASS  │   0    │
│ 4.2.4  │   HIGH   │ Verify that the --read-only-port argument is set to 0                                                           │  PASS  │   0    │
│ 4.2.5  │   HIGH   │ Ensure that the --streaming-connection-idle-timeout argument is not set to 0                                    │  PASS  │   0    │
│ 4.2.6  │   HIGH   │ Ensure that the --protect-kernel-defaults argument is set to true                                               │  PASS  │   0    │
│ 4.2.7  │   HIGH   │ Ensure that the --make-iptables-util-chains argument is set to true                                             │  PASS  │   0    │
│ 4.2.8  │   HIGH   │ Ensure that the --hostname-override argument is not set                                                         │  PASS  │   0    │
│ 4.2.9  │   HIGH   │ Ensure that the --event-qps argument is set to 0 or a level which ensures appropriate event capture             │  PASS  │   0    │
│ 4.2.10 │ CRITICAL │ Ensure that the --tls-cert-file and --tls-private-key-file arguments are set as appropriate                     │  PASS  │   0    │
│ 4.2.11 │ CRITICAL │ Ensure that the --rotate-certificates argument is not set to false                                              │  PASS  │   0    │
│ 4.2.12 │ CRITICAL │ Verify that the RotateKubeletServerCertificate argument is set to true                                          │  PASS  │   0    │
│ 4.2.13 │ CRITICAL │ Ensure that the Kubelet only makes use of Strong Cryptographic Ciphers                                          │  PASS  │   0    │
│ 5.1.1  │   HIGH   │ Ensure that the cluster-admin role is only used where required                                                  │  PASS  │   0    │
│ 5.1.2  │   HIGH   │ Minimize access to secrets                                                                                      │  PASS  │   0    │
│ 5.1.3  │   HIGH   │ Minimize wildcard use in Roles and ClusterRoles                                                                 │  PASS  │   0    │
│ 5.1.6  │   HIGH   │ Ensure that Service Account Tokens are only mounted where necessary                                             │  PASS  │   0    │
│ 5.1.8  │   HIGH   │ Limit use of the Bind, Impersonate and Escalate permissions in the Kubernetes cluster                           │  PASS  │   0    │
│ 5.2.2  │   HIGH   │ Minimize the admission of privileged containers                                                                 │  PASS  │   0    │
│ 5.2.3  │   HIGH   │ Minimize the admission of containers wishing to share the host process ID namespace                             │  PASS  │   0    │
│ 5.2.4  │   HIGH   │ Minimize the admission of containers wishing to share the host IPC namespace                                    │  PASS  │   0    │
│ 5.2.5  │   HIGH   │ Minimize the admission of containers wishing to share the host network namespace                                │  PASS  │   0    │
│ 5.2.6  │   HIGH   │ Minimize the admission of containers with allowPrivilegeEscalation                                              │  FAIL  │   1    │
│ 5.2.7  │  MEDIUM  │ Minimize the admission of root containers                                                                       │  FAIL  │   2    │
│ 5.2.8  │  MEDIUM  │ Minimize the admission of containers with the NET_RAW capability                                                │  PASS  │   0    │
│ 5.2.9  │   LOW    │ Minimize the admission of containers with added capabilities                                                    │  PASS  │   0    │
│ 5.2.10 │   LOW    │ Minimize the admission of containers with capabilities assigned                                                 │  FAIL  │   1    │
│ 5.2.11 │  MEDIUM  │ Minimize the admission of containers with capabilities assigned                                                 │  PASS  │   0    │
│ 5.2.12 │  MEDIUM  │ Minimize the admission of HostPath volumes                                                                      │  PASS  │   0    │
│ 5.2.13 │  MEDIUM  │ Minimize the admission of containers which use HostPorts                                                        │  PASS  │   0    │
│ 5.3.1  │  MEDIUM  │ Ensure that the CNI in use supports Network Policies (Manual)                                                   │   -    │   -    │
│ 5.3.2  │  MEDIUM  │ Ensure that all Namespaces have Network Policies defined                                                        │  PASS  │   0    │
│ 5.4.1  │  MEDIUM  │ Prefer using secrets as files over secrets as environment variables (Manual)                                    │   -    │   -    │
│ 5.4.2  │  MEDIUM  │ Consider external secret storage (Manual)                                                                       │   -    │   -    │
│ 5.5.1  │  MEDIUM  │ Configure Image Provenance using ImagePolicyWebhook admission controller (Manual)                               │   -    │   -    │
│ 5.7.1  │  MEDIUM  │ Create administrative boundaries between resources using namespaces (Manual)                                    │   -    │   -    │
│ 5.7.2  │  MEDIUM  │ Ensure that the seccomp profile is set to docker/default in your pod definitions                                │  FAIL  │   1    │
│ 5.7.3  │   HIGH   │ Apply Security Context to Your Pods and Containers                                                              │  FAIL  │   4    │
│ 5.7.4  │  MEDIUM  │ The default namespace should not be used                                                                        │  PASS  │   0    │
└────────┴──────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────┴────────┘

Operating System

Linux

Version

Version: 0.50.1
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-04-08 12:12:42.137892703 +0000 UTC
  NextUpdate: 2024-04-08 18:12:42.137892442 +0000 UTC
  DownloadedAt: 2024-04-08 14:32:27.417835 +0000 UTC
Policy Bundle:
  Digest: sha256:cdff1bc8c97e4f5cd04782b057c00f5ea8cd81147a506ac4be76bef13710f2d3
  DownloadedAt: 2024-04-08 14:32:27.720489 +0000 UTC

Checklist

• Run trivy image --reset
• Read the troubleshooting

[avaniicf]

Hi @chen-keinan, is there any update to this issue?

[chen-keinan]

as discussed, the severity field will be removed from compliance reports. To match cis specifications

[avaniicf]

There should be some indication of severities that should tell us if there is any HIGH or a CRITICAL finding that needs to be fixed. Without a severity, I am not sure what would be the significance of scanning a kubernetes deployment or a pod using trivy k8s -n prd --compliance=k8s-cis --report summary deployment/xxxx-pod

[avaniicf]

@chen-keinan, the summary report with severities are fine as it displays the highest severity of the IaC checks within a particular control check. The real issue is with the detailed report (report --all) not displaying IaC checks mapped to their compliance control check.
As we now know that the severity filter works only on IaC checks and not on compliance control check severity, hence when you run the scan with report --all displaying the severity for each IaC checks mapped to a compliance control check would fix the issue and make the most sense.

[chen-keinan]

@avaniicf following to our discussion I came to a conclusion that adding severity to compliance report is unnecessary and do not comply to official cis-benchmarks specs as shown.

IaC checks still will have severities and can be assess the cluster with the usual command and you'll get results which you can filter with severity :
trivy k8s cluster --scanners config --severity HIGH

however the compliance cis report IMHO should no include severities as it cause confusion and do not comply to cis-benchmark specifications

[avaniicf]

Hi @chen-keinan, I understand your perspective of not using severity in compliance reports and how it aligns with CIS benchmark specifications.

To clarify further-

1. Are you suggesting a command like trivy k8s deployment/xxxx-pod --scanners misconfig --severity HIGH -n dev for scanning as --scanners config is deprecated?

2. Does scanning with --scanners misconfig include a comprehensive compliance check against the full set of CIS benchmarks or does it focus primarily on detecting misconfigurations?
   Because our ultimate aim is to perform the compliance scan using the CIS benchmark (--compliance=k8s-cis).

[chen-keinan]

@avaniicf --scanners config is the right way, but you'll have to execute it in a cluster scope. In order to get infra assessments

trivy k8s cluster --scannners config --report all --format json

Or you can run it with default scanners and youll get vulns and secrets results as well

trivy k8s cluster --report all --format json

[avaniicf]

@chen-keinan, okay then what about my 2nd question in my previous reply -

Does scanning with --scanners config include a comprehensive compliance check against the full set of CIS benchmarks or does it focus primarily on detecting misconfigurations?
Because our ultimate aim is to perform the compliance scan using the CIS benchmark (--compliance=k8s-cis).