Reconciler fast workload issue.

[kfox1111]

When you have short running jobs on your cluster such as gitlab runner based jobs, it is entirely possible that the job can be created, run, and get destroyed before the vulnerabilityreport job finishes executing. In this case, the starboard reconciler breaks because it can't find the pod anymore.

{
  "level":"error",
  "ts":1636588972.656098,
  "logger":"controller.job",
  "msg":"Reconciler error",
  "reconciler group":"batch",
  "reconciler kind":"Job",
  "name":"scan-vulnerabilityreport-7c49bb98c9",
  "namespace":"starboard-operator",
  "error":"Pod \"runner-sf3jquvm-project-22-concurrent-18pqdh\" not found",
  "stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.10.0/pkg/internal/controller/controller.go:227"
}

Environment:

• Starboard version (use starboard version):
  0.12.0

• Kubernetes version (use kubectl version):
  1.21.5

• OS (macOS 10.15, Windows 10, Ubuntu 19.10 etc):
  starboard-operator

[kfox1111]

We just tried the new trivy-operator with: aquasecurity/trivy-operator#523
excluding just the raw 'pod' workload type. Since gitlab-runners launch bare pods, this seems to have worked around the issue for now. Doing it finer grained with label selectors would be nice.

[chen-keinan]

@kfox1111 thank you the feedback , could you raise an issue for it in trivy-operator repo

[kfox1111]

aquasecurity/trivy-operator#585