Treat a NotFound resource as observed absence, not an unobserved check

[jessie1111101]

Found running deploy-hello-app end to end. The agent never created the namespace, and the objective checking it recorded error, not fail:

namespace-pss-enforced   error   weight 0.3333
  kubectl get namespace failed: command failed with exit code 1: kubectl get namespace hello-app -o json
  stderr: Error from server (NotFound): namespaces "hello-app" not found

Why this is a scoring bug

rollup.py:102 drops an errored entry from both the numerator and the denominator — correctly, since error is defined in base.py:94 as the absence of an observation.

But NotFound is an observation, and an unambiguous one. Scoring it as unobserved means the most common way an agent fails an objective — never creating the resource — quietly shrinks the denominator instead of counting against it. Eleven passes plus one never-created resource reads as 11/11 = 1.0 rather than 11/12. It fails silently, and in the direction nobody audits.

Why it only happens sometimes

resource_property already fails closed on an empty match:

# Fail closed above the flattening. This is what keeps "zero objects
# existed" distinct from "objects existed but the path matched nothing".
if not objects:
    return "fail", f"no {self.kind} matched", raw

That is the intended behaviour. The problem is that absence only reaches that branch one of the two ways it can arrive:

Query	kubectl	Lands on
selector — get ns -l app=x	exit 0, {"items": []}	the fail-closed branch ✅
named — get namespace hello-app	exit 1, NotFound	the blanket except → error ❌

So the same condition scores differently depending on how the objective was written.

The change

is_not_found() lives in k8s/ next to get_resource — recognising kubectl's output is a kubectl concern, while the fail-versus-error policy stays in the verifiers. It matches the apiserver's (NotFound) reason code in full, parentheses included, so bash: kubectl: command not found still errors rather than scoring as an observed absence.

In resource_property the handler substitutes an empty payload and falls through, which reuses the existing semantics rather than duplicating them — so absent still passes when the resource is gone, and every other operator fails.

scaling_complete had the identical hole for its named deployment and is fixed alongside. pod_healthy queries by selector and was never affected.

Tests

Four new cases, verified to actually catch the regression — reverting the fix fails exactly 3 and passes the rest:

• named resource absent → fail (was error)
• absent operator with the resource gone → pass
• connection refused → still error
• command not found → still error (guards the strict marker)

Note the pre-existing test_subprocess_error_is_reported_in_reason uses stderr="not found" and still expects error — it passes unchanged, which is a live check that the strict matching does not over-trigger.

1186 tests pass; ruff and boilerplate clean.

Summary by CodeRabbit

• Bug Fixes

  • Improved handling of Kubernetes resources that do not exist.
  • Absence checks now pass correctly when a resource is missing, while existence checks fail as expected.
  • Missing deployments now produce a clear verification failure instead of an error.
  • Other retrieval failures continue to be reported as errors.

• Tests

  • Added coverage for missing resources, deployments, and unrelated subprocess errors.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jessie1111101
Once this PR has been reviewed and has the lgtm label, please assign janetkuo for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

Hi @jessie1111101. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Tip

We noticed you've done this a few times! Consider joining the org to skip this step and gain /lgtm and other bot rights. We recommend asking approvers on your previous PRs to sponsor you.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

The change adds exact kubectl NotFound detection and applies it to resource property and scaling verification. Tests cover missing resources, missing deployments, unrelated subprocess failures, and misleading “not found” messages.

Changes

Kubernetes NotFound handling

Layer / File(s)	Summary
NotFound detection and exports devops_bench/k8s/kubectl.py, devops_bench/k8s/__init__.py	Adds is_not_found and exports it from the Kubernetes package. The helper matches kubectl’s exact (NotFound) stderr marker.
Verifier error handling and tests devops_bench/verification/verifiers/resource_property.py, devops_bench/verification/verifiers/scaling_complete.py, tests/unit/verification/test_resource_property.py, tests/unit/verification/test_scaling_complete.py	Resource absence produces an empty item list for property checks. Missing deployments produce a failed verification with a not-found reason. Other subprocess failures remain errors. Unit tests cover these cases.

Estimated code review effort: 2 (Simple) | ~15 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: treating Kubernetes NotFound resources as observed absence during verification.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}