feat: implement issue #605 — Shadow-mode dual-run for safe agent canary validation

.github/workflows/lint.yml

@@ -94,6 +94,7 @@
             tests/test_model_ab.bats \
             tests/test_triage_prompt.bats \
             tests/test_holdout_guard.bats \
+            tests/test_shadow_compare.bats \
             tests/test_skill_eval_report.bats \
             tests/test_eval_health_notify.bats \
             tests/test_consumer_manifest.bats \
@@ -187,7 +188,7 @@
     timeout-minutes: 5
     steps:
       - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10  # v6.0.3
         # NOTE: persist-credentials is intentionally left at the default (true) —
         # vci_resolve_reusable needs `git fetch origin` to authenticate against this
         # private repo when resolving same-repo channel tags. Setting
         # persist-credentials: false would silently push every such caller into the

docs/release/runbook.md

@@ -167,7 +167,10 @@ git push --force origin dev-lead/stable
   immutable `vX.Y.Z` (§3) — for that ring's channel, or for `stable` if already
   promoted.
 - The fully automated, health-gated version of this loop is issue #501; today it
-  is a human-driven sequence of the moves above.
+  is a human-driven sequence of the moves above. A required input to that gate is
+  the **shadow-mode dual-run** signal (#605) — run the `next` candidate silently
+  alongside `stable` on a PR and compare, blocking promotion on a regression. See
+  [`shadow-mode.md`](./shadow-mode.md).
 
 Then **verify** (§4).
 

docs/release/shadow-mode.md

@@ -0,0 +1,88 @@
+# Shadow-mode dual-run — safe agent canary validation
+
+Shadow-mode is a health-gate signal for the per-agent channel-tag release model
+(initiative [#495](../initiatives/agentic-release-strategy.md)). It is issue
+**#605**, promoted from Ideas Discussion #566 so the dependency is a real
+`blocked_by` edge: it is a **prerequisite for the self-improving-skills proposer
+(#587)** and a peer of health-gated promotion (#501).
+
+## What it does
+
+On a single PR, two agent lanes run in parallel:
+
+- the **`stable` lane** — production duty; its output is posted to the PR;
+- the **`next` (shadow) lane** — the candidate release, run **silently**: its
+  output is **never posted** to the PR, only logged.
+
+The shadow output is compared against the stable output to detect a
+**quality regression** — the candidate doing demonstrably worse than the version
+it would replace on the same input. The comparison yields a machine-readable
+signal (`shadow_dual_run`) that the health-gated promotion gate (#501) consumes
+as a **required** input before advancing a candidate ring.
+
+This directly serves the initiative: it validates a `next` candidate against live
+production behaviour before any consumer sees it (SC5), without exposing the PR to
+the unvalidated candidate (only `stable` posts).
+
+## Signal contract
+
+The comparison is pure logic in [`scripts/lib/shadow-compare.sh`](../../scripts/lib/shadow-compare.sh);
+the wrapper [`scripts/shadow-run.sh`](../../scripts/shadow-run.sh) reads the two
+lanes' results and emits the signal. `sc_classify` returns one status:
+
+| Status | Meaning | Blocks promotion? |
+|---|---|---|
+| `MATCH` | Both lanes succeed, outputs equal after normalization | No — strongest healthy signal |
+| `DIVERGED` | Both succeed but outputs differ | **No — advisory only** |
+| `REGRESSION` | Stable succeeds, shadow does not (failed / empty / errored) | **Yes** |
+| `SHADOW_ONLY_OK` | Stable did not succeed but shadow did (candidate may be a fix) | No |
+| `BOTH_FAILED` | Neither succeeded (environmental / PR-specific) | No — inconclusive |
+| `NO_SHADOW` | No shadow run observed | No — inconclusive |
+
+**Only `REGRESSION` blocks promotion.** `DIVERGED` is advisory because review
+quality is not objectively measurable — A/B quality routing is deferred (see the
+initiative analysis §5 / Option D). The gate therefore halts a candidate only when
+it is *provably* worse than the version it replaces, and logs a divergence for
+human review otherwise.
+
+The emitted JSON signal (written to `SHADOW_SIGNAL_OUT`, surfaced in `GITHUB_ENV`
+as `SHADOW_STATUS` / `SHADOW_BLOCK_PROMOTION`):
+
+```json
+{
+  "signal": "shadow_dual_run",
+  "reusable": "dev-lead",
+  "channel": "next",
+  "status": "REGRESSION",
+  "regression": true,
+  "blocks_promotion": true,
+  "stable_run_id": 111,
+  "shadow_run_id": 222
+}
+```
+
+`shadow-run.sh` always exits 0 for a completed comparison — the shadow lane must
+never disrupt the PR. A regression surfaces as a `::warning::` plus the env flags
+and signal artifact, not as a failed required check.
+
+## Wiring into promotion (#501)
+
+`shadow_dual_run` is declared as a required gate signal in
+[`release/registry.yml`](../../release/registry.yml) under
+`reusables.dev-lead.gate.signals`. The Release_Manager soak-and-promote loop
+(#993/#999) reads this registry; when #501's health gate evaluates a ring it must
+confirm the `shadow_dual_run` signal is non-blocking before advancing (see the
+staged rollout in [`runbook.md` §2c](./runbook.md#2c-staged-canary--ring-rollout)).
+
+## Integration status — the dispatch half
+
+This change delivers the **comparison + signal** half of shadow-mode: given the
+two lanes' results, it classifies and emits the promotion signal. The remaining
+**dispatch** half — running the `next` lane in parallel with `stable` on the same
+PR while suppressing the shadow's PR output — requires a *silent/shadow-mode input*
+on the org-canonical `dev-lead-reusable.yml` (and the `pr-review` reusable), which
+AGENTS.md restricts from ad-hoc modification. That reusable input plus the workflow
+that dispatches both lanes and feeds their run ids/outputs into `shadow-run.sh` is
+the follow-up integration step; this issue lands the signal contract the gate
+depends on, mirroring how `release/registry.yml` was landed ahead of its #501
+reader.

release/registry.yml

@@ -11,7 +11,11 @@
 #                   cross-repo and this repo holds only the thin caller stub).
 #     run_workflow  the workflow (file or display name) whose run history feeds the
 #                   health gate for this reusable.
-#     gate          per-reusable gate knobs (soak_window_days, ...).
+#     gate          per-reusable gate knobs (soak_window_days, signals, ...).
+#                   signals is the list of required health signals the gate must
+#                   confirm before promoting a candidate ring. `shadow_dual_run`
+#                   (#605) is the shadow-mode comparison signal emitted by
+#                   scripts/shadow-run.sh — see docs/release/shadow-mode.md.
 #     rings         ordered rollout ladder — a list of { channel, order, repos }.
 #                   A candidate soaks in each ring in `order` and is promoted to the
 #                   next only once the gate confirms the current ring is healthy.
@@ -29,7 +33,7 @@ reusables:
   dev-lead:
     host: petry-projects/.github-private
     run_workflow: dev-lead.yml
-    gate: { soak_window_days: 7 }
+    gate: { soak_window_days: 7, signals: [shadow_dual_run] }
     rings:
       - { channel: next,   order: 0, repos: [petry-projects/.github-private] }
       - { channel: ring0,  order: 1, repos: [petry-projects/.github] }

scripts/lib/shadow-compare.sh

@@ -0,0 +1,182 @@
+#!/usr/bin/env bash
+# shadow-compare.sh — shadow-mode dual-run comparison + promotion signal (#605).
+#
+# Shadow-mode runs the `next`-channel agent in parallel with the `stable` agent on
+# the same PR: only `stable`'s output is posted to the PR, while `next` runs as a
+# silent shadow whose output is logged and compared against `stable`. The
+# comparison yields a quality-regression signal that the health-gated promotion
+# gate (#501) consumes as a required input before advancing a candidate ring.
+# This unblocks the self-improving-skills proposer (#587), which gates on a safe
+# canary; it is part of the Safe Release Strategy (#495).
+#
+# This library is PURE — sourced by the wrapper (scripts/shadow-run.sh) and by
+# tests. No network, no agent dispatch, no side effects. The wrapper owns I/O.
+#
+# Classification (sc_classify), stable-vs-shadow on the same PR:
+#   MATCH          both succeed, outputs equal after normalization — strongest
+#                  healthy signal (candidate reproduces production exactly)
+#   DIVERGED       both succeed but outputs differ — ADVISORY only. Review quality
+#                  is not objectively measurable (A/B quality routing is deferred,
+#                  see docs/initiatives/agentic-release-strategy.md §5/Option D),
+#                  so a divergence is logged for human review but does not block.
+#   REGRESSION     stable succeeds but shadow does not (failed/empty/errored) —
+#                  the candidate did demonstrably worse than production on the same
+#                  input. This is the one BLOCKING signal for promotion.
+#   SHADOW_ONLY_OK stable did not succeed but shadow did — the candidate may be a
+#                  fix; do not block promotion on an unrelated stable failure.
+#   BOTH_FAILED    neither succeeded — environmental / PR-specific, not attributable
+#                  to the candidate. Inconclusive, non-blocking.
+#   NO_SHADOW      no shadow run was observed — inconclusive, non-blocking.
+#
+# Only REGRESSION blocks promotion (sc_is_blocking); every other status is
+# healthy, advisory, or inconclusive. This keeps the gate conservative: it halts a
+# candidate only when it is provably worse than the version it would replace.
+
+SC_SIGNAL_TYPE="shadow_dual_run"
+
+# sc_normalize <text> — strip leading/trailing whitespace (incl. newlines) so two
+# outputs differing only by surrounding whitespace compare equal. Pure.
+sc_normalize() {
+  local text="${1:-}"
+  # Trim leading whitespace (including newlines), then trailing whitespace.
+  text="${text#"${text%%[![:space:]]*}"}"
+  text="${text%"${text##*[![:space:]]}"}"
+  printf '%s' "$text"
+}
+
+# sc_conclusion_ok <conclusion> — return 0 iff the run concluded success.
+sc_conclusion_ok() {
+  [ "${1:-}" = "success" ]
+}
+
+# sc_conclusion_present <conclusion> — return 0 iff a run was actually observed.
+# Empty/"null" means no run happened; "skipped" means the lane was gated off
+# (conditional/paths filter) and never executed — both are "no shadow run", so
+# they classify as NO_SHADOW (inconclusive), NOT a REGRESSION false-positive.
+sc_conclusion_present() {
+  case "${1:-}" in
+    ""|null|skipped) return 1 ;;
+    *)               return 0 ;;
+  esac
+}
+
+# sc_classify <stable_conclusion> <shadow_conclusion> <stable_out> <shadow_out>
+# Echo one status token (see header). Pure.
+sc_classify() {
+  local stable_concl="${1:-}" shadow_concl="${2:-}"
+  local stable_out="${3:-}" shadow_out="${4:-}"
+
+  # No shadow run at all — nothing to compare, inconclusive.
+  if ! sc_conclusion_present "$shadow_concl"; then
+    printf '%s' "NO_SHADOW"
+    return 0
+  fi
+
+  local stable_ok=1 shadow_ok=1
+  sc_conclusion_ok "$stable_concl" && stable_ok=0
+  sc_conclusion_ok "$shadow_concl" && shadow_ok=0
+
+  if [ "$stable_ok" -eq 0 ] && [ "$shadow_ok" -eq 0 ]; then
+    # Both concluded success — compare the actual outputs.
+    local sn hn
+    sn="$(sc_normalize "$stable_out")"
+    hn="$(sc_normalize "$shadow_out")"
+    # A shadow that "succeeds" but emitted nothing where stable produced output is
+    # a silent regression, not a match.
+    if [ -z "$hn" ] && [ -n "$sn" ]; then
+      printf '%s' "REGRESSION"
+    elif [ "$sn" = "$hn" ]; then
+      printf '%s' "MATCH"
+    else
+      printf '%s' "DIVERGED"
+    fi
+    return 0
+  fi
+
+  if [ "$stable_ok" -eq 0 ] && [ "$shadow_ok" -ne 0 ]; then
+    printf '%s' "REGRESSION"
+    return 0
+  fi
+
+  if [ "$stable_ok" -ne 0 ] && [ "$shadow_ok" -eq 0 ]; then
+    printf '%s' "SHADOW_ONLY_OK"
+    return 0
+  fi
+
+  printf '%s' "BOTH_FAILED"
+}
+
+# sc_is_blocking <status> — return 0 iff the status must block promotion. Only a
+# REGRESSION blocks; every other status is healthy/advisory/inconclusive.
+sc_is_blocking() {
+  [ "${1:-}" = "REGRESSION" ]
+}
+
+# sc_signal_json <status> <reusable> <channel> <stable_run_id> <shadow_run_id>
+# Emit the compact JSON signal the #501 promotion gate reads. Run ids are numeric
+# when present and null when absent. Pure (uses jq for correct escaping/typing).
+sc_signal_json() {
+  local status="${1:-}" reusable="${2:-}" channel="${3:-}"
+  local stable_run_id="${4:-}" shadow_run_id="${5:-}"
+  local blocking=false regression=false
+  if sc_is_blocking "$status"; then
+    blocking=true
+    regression=true
+  fi
+
+  jq -cn \
+    --arg signal "$SC_SIGNAL_TYPE" \
+    --arg reusable "$reusable" \
+    --arg channel "$channel" \
+    --arg status "$status" \
+    --argjson regression "$regression" \
+    --argjson blocks "$blocking" \
+    --arg stable_run_id "$stable_run_id" \
+    --arg shadow_run_id "$shadow_run_id" \
+    '{
+      signal: $signal,
+      reusable: $reusable,
+      channel: $channel,
+      status: $status,
+      regression: $regression,
+      blocks_promotion: $blocks,
+      stable_run_id: (($stable_run_id | tonumber?) // null),
+      shadow_run_id: (($shadow_run_id | tonumber?) // null)
+    }'
+}
+
+# sc_report <status> <reusable> <channel> <stable_run_url> <shadow_run_url> [today]
+# Markdown body for the workflow log / step summary. This is NEVER posted to the
+# PR — only stable's output reaches the PR; the shadow lane is silent. Pure.
+sc_report() {
+  local status="${1:-}" reusable="${2:-}" channel="${3:-}"
+  local stable_url="${4:-}" shadow_url="${5:-}" today="${6:-}"
+  [ -n "$today" ] || today="$(date -u +%Y-%m-%d)"
+
+  local icon headline
+  case "$status" in
+    MATCH)          icon='✅'; headline='shadow reproduced stable exactly' ;;
+    DIVERGED)       icon='🟡'; headline='shadow diverged from stable (advisory — logged for human review)' ;;
+    REGRESSION)     icon='🔴'; headline='shadow did worse than stable (blocks promotion)' ;;
+    SHADOW_ONLY_OK) icon='🟢'; headline='shadow succeeded where stable did not (candidate may be a fix)' ;;
+    BOTH_FAILED)    icon='⚪'; headline='both lanes failed (inconclusive — likely environmental)' ;;
+    NO_SHADOW)      icon='⚪'; headline='no shadow run observed (inconclusive)' ;;
+    *)              icon='⚪'; headline="unknown status: ${status}" ;;
+  esac
+
+  printf '# %s Shadow-mode dual-run — %s\n\n' "$icon" "$today"
+  printf '_Reusable `%s` · candidate channel `%s` · signal `%s`_\n\n' \
+    "$reusable" "$channel" "$SC_SIGNAL_TYPE"
+  printf -- '- **Result:** %s — %s\n' "$status" "$headline"
+  if sc_is_blocking "$status"; then
+    printf -- '- **Promotion:** BLOCKED — this is a required health-gate signal (#501).\n'
+  else
+    printf -- '- **Promotion:** not blocked by this signal.\n'
+  fi
+  [ -n "$stable_url" ] && printf -- '- **Stable run:** %s\n' "$stable_url"
+  [ -n "$shadow_url" ] && printf -- '- **Shadow run:** %s\n' "$shadow_url"
+  printf '\n'
+  printf '> The shadow (`%s`) output is **not posted** to the PR — only the `stable` ' "$channel"
+  printf 'lane posts. The shadow output is logged and compared here for '
+  printf 'quality-regression detection feeding health-gated promotion.\n'
+}