node-collector not scheduling

[yuvals41]

Description

Description: I encountered an issue while running the following Trivy command:

trivy k8s --tolerations nodeGroupe=gp-mix:NoSchedule --exclude-nodes nodeGroup:tcp,nodeGroup:asm,nodeGroup:attack --severity=CRITICAL --report=summary --debug

The job creates pods, but they get stuck in a loop where they run, complete, terminate, and then restart, eventually staying in Pending status. Here is the error message I get in the logs:

2024-10-14T13:51:44+03:00	FATAL	Fatal error
  - get k8s artifacts with node info error:
    github.com/aquasecurity/trivy/pkg/k8s/commands.clusterRun
        /home/runner/work/trivy/trivy/pkg/k8s/commands/cluster.go:42
  - running node-collector job: runner received timeout

When inspecting the pods, I can see the following pattern:

node-collector-55ddcd9f6f-k4wrh   0/1     Pending   0          0s
node-collector-55ddcd9f6f-k4wrh   1/1     Running   0          1s
node-collector-55ddcd9f6f-k4wrh   0/1     Completed 0          7s
node-collector-55ddcd9f6f-k4wrh   0/1     Terminating 0        10s
node-collector-68d9bc8b64-dg6v4   0/1     Pending   0          0s

From the pod's description, there are FailedScheduling events due to node taints, such as nodeGroup: tcp, nodeGroup: asm, and nodeGroup: gp-mix.

Desired Behavior

The job should create a pod that runs, completes its task, and exits properly without restarting or staying in a pending loop.

Actual Behavior

The node-collector pods are running, completing, and restarting multiple times, eventually leading to a Pending state, where the pods are unable to be scheduled on any node due to taint issues.

Reproduction Steps

1. Run the following command:
 k8s --tolerations nodeGroupe=gp-mix:NoSchedule --exclude-nodes nodeGroup:tcp,nodeGroup:asm,nodeGroup:attack --severity=CRITICAL --report=summary --debug

2. Check the logs and the pod status under the trivy-temp namespace
3. Observe the repeated creation, completion, and termination of the pods.
...

Target

Kubernetes

Scanner

Vulnerability

Output Format

Table

Mode

Standalone

Debug Output

2024-10-14T13:51:44+03:00	FATAL	Fatal error
  - get k8s artifacts with node info error:
    github.com/aquasecurity/trivy/pkg/k8s/commands.clusterRun
        /home/runner/work/trivy/trivy/pkg/k8s/commands/cluster.go:42
  - running node-collector job: runner received timeout

Operating System

I'm running Trivy on ubuntu 22.04.1

Version

Version: 0.56.2
Vulnerability DB:
  Version: 2
  UpdatedAt: 2024-10-14 00:19:40.198722045 +0000 UTC
  NextUpdate: 2024-10-15 00:19:40.198721795 +0000 UTC
  DownloadedAt: 2024-10-14 09:34:44.561344051 +0000 UTC
Check Bundle:
  Digest: sha256:ae151c4eecf35c507d8f866121ddfbf46540b041bc7bca7cdd8d9f70ceb6f12c
  DownloadedAt: 2024-10-14 08:15:29.018928598 +0000 UTC

Checklist

• Run trivy clean --all
• Read the troubleshooting

[afdesk]

@yuvals41 thanks for the report!

What is your k8s version?
it's a known situation for v1.31.1 - #7705

[yuvals41]

Thanks for the response!

my kubernetes version is 1.30, its an eks cluster

[afdesk]

thanks for the details. It's interesting
I'll take a look at this version too

[yuvals41]

Thanks!

[afdesk]

It takes some time. I'll update you in a few days