Bug Description
Diagnostic summary (feedback/bug-report)
Title: Inverted absence-of-evidence search result → shipped a change that would have broken all production task execution paths
Severity: High (caught pre-merge by external review; would have zeroed IsMachineProvisioner/IsDocker/IsProvisionerRemoteDocker/ExternalLifecycle/DisableSpinUpStep/DisableIsolatedSSHDir for every task across machine-provisioner, classic docker, remote-docker, and the k8s operator)
What happened: Claimed 6 build-agent flags were "CLI-only by convention" based on (a) MarkHidden status and (b) absence from one local test's fixture — neither entails the claim. Ran a GitHub code search for the CLI flag name strings across the org; it returned zero hits outside the flag's own definition. Reported this null result as confirming CLI-only usage, when the correct reading is the opposite: no caller invokes the CLI flag at all, so (combined with the fields having real JSON tags) they must be set via the config file instead. Shipped a PR (json:"-" on all 6 fields) built entirely on this inverted conclusion, without reading the actual caller source first.
Root cause: Absence-of-evidence for hypothesis A was misread as confirmation of hypothesis A, rather than as a signal to directly test hypothesis B (JSON-only usage). Search was also asymmetric — it could only detect CLI usage strings, not JSON key strings — so it structurally couldn't have falsified the standing hypothesis either way.
Contributing factors:
- Grounded the initial hypothesis in properties (MarkHidden, one test's coverage) that don't imply the conclusion drawn.
- Substituted a cheap proxy signal (grep hit count) for ground truth (reading agent/task/runner.go, agent/internal/docker/run.go, operator/config.go) that was one clone-and-read away.
- Verification effort didn't scale with blast radius — a claim gating a change to every task-execution path across 3 executor types was accepted on a single ambiguous grep.
- Confident language ("Confirmed via search") was applied to a weak, actually-contradictory signal, and that confidence then propagated unchallenged through branch name, commit message, PR body, and code comments — each restatement looked like independent corroboration but wasn't.
- Adversarial code review gave a false sense of closure ("0 findings") despite being structurally unable to catch this class of bug — it only sees the local diff, never the calling repos' actual invocation code.
Corrective action taken: Traced all three real call sites directly; confirmed 6 fields are JSON-config-only and 1 (DeleteConfigAfterRead) is genuinely CLI-only. Corrected PR #4083 to the verified scope, added a regression test locking in the correct JSON-settable behavior for the 6 fields, corrected the record on both PRs (#4077, #4083).
Preventive measure: Saved as feedback_cross_repo_call_site_verification.md — cross-repo behavioral claims require reading actual call sites in every caller, not inference from a code-search null result; treat "zero hits" as a prompt to test the alternate hypothesis, not as confirmation of the standing one.
Environment Info
- Platform: darwin
- Terminal: iTerm.app
- Version: 2.1.236
- Feedback ID: d5ed014f-0695-4925-8a85-40ba3b705613
Errors
Bug Description
Diagnostic summary (feedback/bug-report)
Title: Inverted absence-of-evidence search result → shipped a change that would have broken all production task execution paths
Severity: High (caught pre-merge by external review; would have zeroed IsMachineProvisioner/IsDocker/IsProvisionerRemoteDocker/ExternalLifecycle/DisableSpinUpStep/DisableIsolatedSSHDir for every task across machine-provisioner, classic docker, remote-docker, and the k8s operator)
What happened: Claimed 6 build-agent flags were "CLI-only by convention" based on (a) MarkHidden status and (b) absence from one local test's fixture — neither entails the claim. Ran a GitHub code search for the CLI flag name strings across the org; it returned zero hits outside the flag's own definition. Reported this null result as confirming CLI-only usage, when the correct reading is the opposite: no caller invokes the CLI flag at all, so (combined with the fields having real JSON tags) they must be set via the config file instead. Shipped a PR (json:"-" on all 6 fields) built entirely on this inverted conclusion, without reading the actual caller source first.
Root cause: Absence-of-evidence for hypothesis A was misread as confirmation of hypothesis A, rather than as a signal to directly test hypothesis B (JSON-only usage). Search was also asymmetric — it could only detect CLI usage strings, not JSON key strings — so it structurally couldn't have falsified the standing hypothesis either way.
Contributing factors:
Corrective action taken: Traced all three real call sites directly; confirmed 6 fields are JSON-config-only and 1 (DeleteConfigAfterRead) is genuinely CLI-only. Corrected PR #4083 to the verified scope, added a regression test locking in the correct JSON-settable behavior for the 6 fields, corrected the record on both PRs (#4077, #4083).
Preventive measure: Saved as feedback_cross_repo_call_site_verification.md — cross-repo behavioral claims require reading actual call sites in every caller, not inference from a code-search null result; treat "zero hits" as a prompt to test the alternate hypothesis, not as confirmation of the standing one.
Environment Info
Errors