[runtime-observer] Skip unavailable Helix evidence

[vitek-karas]

Motivation

The runtime-failure observer can encounter Helix evidence that is no longer retained or cannot be retrieved for an individual candidate. Treating these expected evidence gaps as a fatal scan failure prevents the observer from continuing to other builds and work items, even when those later candidates have actionable xharness failures. The change distinguishes evidence-retention gaps from malformed responses and actual request failures.

Changes

• Skip empty logs, missing Helix job identifiers, unavailable work items, and Helix console HTTP 404s on a per-candidate basis.
• Keep malformed or unexpectedly shaped payloads, unreadable or binary logs, access failures, authentication failures, timeouts, and unexplained HTTP errors fatal.
• Validate work-item names before console requests and add focused tests for the new helper error paths.
• Regenerate the workflow lock metadata.

Validation

• gh aw compile .github/workflows/runtime-failure-observer.agent.md --no-emit
• python3 .github/workflows/tests/test_runtime_failure_observer_http.py RequestBehaviorTests HelixTraversalTests

[Copilot]

🟡 Changes recommended

Two moderate findings remain unresolved in the Helix validation and skip handling.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Updates the runtime observer to skip unavailable Helix evidence while keeping malformed data and request failures fatal.

Changes:

• Adds per-candidate Helix skip handling and request-specific error labeling.
• Adds focused HTTP and traversal tests.
• Regenerates workflow lock metadata.

File summaries

File	Reviewed changes and findings
.github/workflows/tests/test_runtime_failure_observer_http.py	Adds focused HTTP and traversal tests.
.github/workflows/runtime-failure-observer.agent.md	Defines evidence-retention handling. Moderate, 2 votes: `
.github/workflows/runtime-failure-observer.agent.lock.yml	Updates generated workflow metadata.
.github/workflows/runtime-failure-observer-http	Adds request-specific HTTP error labels.

Review details

Suppressed comments (1)

.github/workflows/runtime-failure-observer.agent.md:258

• This helper message is not sufficient evidence of retention: the jq expression above passes the literal null when the selected object lacks Name/WorkItemName (or uses a different casing, despite the helper's case-insensitive lookup), and _console_url then reports found 0. The new rule would therefore skip malformed work-item data, contrary to rule 6; validate the selected object/index and require a non-empty name before allowing this skip, keeping missing-field cases fatal.

- The helper reports `expected exactly one Helix work item named '<name>', found 0`: `skipped: Helix work item unavailable`.

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (5)

.github/workflows/runtime-failure-observer.agent.md:304

• This message is not sufficient evidence that the work item is merely unavailable: _console_url also emits found 0 for malformed records. For example, a selected item with a truthy non-string Name and a valid WorkItemName passes the new jq validation, but the helper's Name or WorkItemName expression uses the invalid Name, finds no match, and is now incorrectly downgraded to a skip instead of rule 6's fatal malformed-data path. Align the helper's name selection with the validation (or validate the response shape) before treating found 0 as an evidence-retention signal.

- The helper reports `expected exactly one Helix work item named '<name>', found 0`: `skipped: Helix work item unavailable`.
- The helper reports `Helix console request failed with status 404`: `skipped: Helix console unavailable (HTTP 404)`.

These skips apply only after the helper request itself succeeded far enough to produce the explicit signal. Do not skip malformed or unexpectedly shaped JSON, unreadable or non-text logs, missing required fields, access denials, authentication failures, timeouts, HTTP 401/403/429/5xx, generic transport errors, or any other unexplained HTTP status; apply rule 6 and stop the run. A 404 from the Helix work-items request is not the console 404 signal and is fatal under rule 6.

.github/workflows/runtime-failure-observer.agent.md:241

• pipefail does not preserve the first command's status when the downstream grep exits 1: if the first grep fails with status 2 (for example, a read error or a race after the file check), the second grep sees no input and returns 1, so the pipeline is classified as the expected no-match skip. Capture/check each command's status separately, or use a single extraction that preserves non-1 failures, so malformed/unreadable evidence remains fatal as required by rule 6.

grep -oE 'Sent Helix Job(: |; see work items at https://helix\.dot\.net/api/jobs/)[a-f0-9-]+' /tmp/gh-aw/agent/helix-send.log \
  | grep -oE '[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}'

.github/workflows/runtime-failure-observer.agent.md:233

• This example puts the helper request and its validation in one shell block without preserving the helper's exit status. If a later build's azdo-log fails after an earlier candidate left a readable helix-send.log, the test succeeds and the combined call reports success, allowing stale evidence to be parsed. Rule 11 requires each helper request and follow-up validation to be separate shell calls; show them as separate calls (or otherwise fail on the request error) so transport failures remain fatal.

test -f /tmp/gh-aw/agent/helix-send.log && test -r /tmp/gh-aw/agent/helix-send.log

.github/workflows/runtime-failure-observer.agent.md:269

• The jq normalization only accepts a top-level .value array, but the helper's _json_items contract also accepts WorkItems and workItems (lines 425-431). A valid response in either of those shapes will fail this new preflight with “not an array” and stop the scan, even though helix-console could resolve it; preserve all response shapes supported by the helper here.

  (if type == "array" then . else .value end) as $items
  | if ($items | type) == "array" then $items[INDEX] else error("Helix work-item response is not an array") end

.github/workflows/runtime-failure-observer.agent.md:292

• The console-fetch jq repeats the same .value-only normalization, so responses using the WorkItems or workItems forms accepted by _json_items are rejected after the preflight (or if the preflight is adjusted). Keep this extraction expression consistent with the helper's supported response shapes.

  (if type == "array" then . else .value end)
  | .[INDEX]

• Files reviewed: 4/4 changed files
• Comments generated: 2
• Review effort level: Lite