[Incident] Hourly Product Development blocks its own GitHub API egress

[seonghobae]

Incident summary

The original failure was a Harden Runner egress parsing defect in the hourly product-development workflow, not an NVIDIA credential failure. Protected-main repairs in PRs #23 and #24 removed the observed GitHub API connection-refused failure while preserving exact egress allowlists, NVIDIA_NIM_API_KEY isolation, credential-free reverification, publisher separation, and the one-proposal authority boundary.

Protected-main evidence already satisfied

Scheduled Hourly Product Development run 31343380043 executed on protected main SHA fb7dab5698ffd24b1a6db0943f1e387f0eda4d31 and completed successfully. The deterministic gh api pull-request gate passed through Harden Runner and exited cleanly with reason=open_pull_request; model, reverification, and publication jobs were correctly skipped because the repository had open PRs.

Therefore closure criteria 1–4 are satisfied:

1. exact-head repair CI/actionlint/security/review gates passed;
2. repair reached protected main;
3. a protected-main scheduled execution passed Harden Runner + GitHub API access;
4. with an open PR queue, the scheduler exited cleanly at the documented open_pull_request gate.

The canonical documentation graph subsequently reached protected main through PR #25 merge fe9b46f5404f368b311de205c4e647f47db89ab3; PR #29 post-integration audit merged as 59eb6d609a6cbf0699e427e913067d8b0c3beb41.

Remaining closure criterion

Keep this issue open for criterion 5 only:

• after the PR queue is genuinely drained and release policy permits product development, a protected-main scheduled/manual run must reach the bounded OpenCode/NVIDIA path and either:
  • produce one sealed proposal that passes independent credential-free reverification and the existing publisher boundary, or
  • exit at a documented product/release gate without weakening the deterministic trust boundary.

Do not manufacture an empty PR queue, close valid work prematurely, bypass release blocker #17, broaden egress, expose the NVIDIA key to repository-controlled code, or use COPILOT_GITHUB_TOKEN merely to obtain this proof.

Current dependency order — 2026-08-10

1. PR fix(release): fail before side effects when Trusted Publishing is unready #30 must finish its exact-head CI/security/review/merge cycle. It adds the repository-side Trusted Publishing release-readiness preflight and ADR-0008.
2. Release blocker Release operations: complete PyPI Trusted Publishing for 0.2.0 #17 must then complete the external GitHub pypi environment + PyPI Trusted Publisher + public 0.2.0 acceptance path.
3. Draft PR feat: add incremental mailbox threading with stable identity handoff #20 remains intentionally non-mergeable until Release operations: complete PyPI Trusted Publishing for 0.2.0 #17 closes; after publication it must be refreshed onto the released protected main and fully revalidated.
4. Only after the truthful product/documentation PR queue is drained and release policy allows model-backed development can criterion 5 execute without violating the product freeze or independent review boundaries.

Queued CI/reviewer/provider time on one item is not an incident blocker for other safe repository work; criterion 5 itself is dependency-gated by the truthful empty-queue/release state above.

Closure evidence required

Close this issue only after criterion 5 records:

• exact protected source SHA;
• workflow run/job identities;
• Harden Runner and model-provider boundary outcome;
• bounded OpenCode proposal/defer outcome;
• independent credential-free reverification result when a patch is proposed;
• publisher/defer result;
• confirmation that no credential, egress, review-authority, or one-proposal boundary was weakened to obtain the proof.

[seonghobae]

Operational revalidation after #23 reached protected main found the incident still open. Scheduled Hourly Product Development run 31250354848 at protected-main commit e9e52e4cae47c1c0164a37ed907719f5e899caf5 failed before the deterministic open-PR exit. Harden Runner's workflow input contained both api.github.com:443 and cafe.github.com:443, but its installed agent retained api.github.com with port 0 plus the dynamically space-appended cache host; gh api then resolved api.github.com to 140.82.116.5, the agent dropped that IP, and the request failed with connection refused. This is still not an NVIDIA credential failure.

Follow-up PR #24 is the bounded test-first repair. It preserves egress-policy: block and the exact endpoint sets, but changes the three allowed-endpoints scalars from literal | to folded >- so the action receives a space-delimited runtime string consistent with the pinned action's own dynamic endpoint append behavior and the observed agent parser. Exact-head CI/actionlint, SAST, and Security Scan are currently green on 71a6d7e151ec6a9d53f9761ad65d4f22d3951c5f; independent formal review remains separate. Keep this issue open through protected-main operational reruns required by the existing closure criteria.

[seonghobae]

Protected-main operational acceptance checkpoint — 2026-08-10

Scheduled Hourly Product Development run 31343380043 executed on protected main SHA fb7dab5698ffd24b1a6db0943f1e387f0eda4d31 and completed successfully. The hardened runner accepted both api.github.com:443 and the observed cafe.github.com:443 alias, and the deterministic gh api pull-request gate completed without the prior connection-refused failure. Because open pull requests existed, the gate exited cleanly with reason=open_pull_request; the model, reverification, and publication jobs were correctly skipped.

This satisfies closure criteria 1–4 for the incident. Keep the issue open for criterion 5 only: after the PR queue is genuinely drained (and release policy permits product development), a protected-main scheduled/manual run must reach the bounded OpenCode/NVIDIA path and either publish one independently verified proposal or exit at a documented product gate. Do not manufacture an empty PR queue or bypass release blocker #17 merely to satisfy that proof.

[seonghobae]

Fresh causal-chain and feasibility checkpoint — 2026-08-12

The original Harden Runner incident remains repaired and operationally proven through the open-PR branch. Protected-main evidence already shows the folded endpoint scalar preserves port 443, GitHub API inventory succeeds under fail-closed egress, and the workflow exits open_pull_request before touching the model credential.

The remaining zero-PR acceptance condition is currently blocked by a distinct, legitimate release dependency rather than by egress or NVIDIA authentication:

1. release blocker Release operations: complete PyPI Trusted Publishing for 0.2.0 #17 remains open;
2. repository-side release preflight PR fix(release): fail before side effects when Trusted Publishing is unready #30 is now merged on protected main at 2559425084389176870eac9d1a855d219bc12ce3;
3. GitHub still reports no repository environment named pypi (total_count: 0);
4. the PyPI Trusted Publisher and protected environment review policy therefore cannot yet be exercised;
5. Draft PR feat: add incremental mailbox threading with stable identity handoff #20 must remain behind the release boundary, so the queue cannot truthfully become zero for the final Hourly Product Development acceptance run.

Feasibility decision

Creating another token, removing the release blocker, force-merging #20, weakening the environment preflight, or claiming a model-path run while the queue is intentionally nonzero would not repair the causal chain. The realistic next action is the account-side pypi environment and PyPI Pending Trusted Publisher configuration documented on #17, followed by the protected 0.2.0 release, #20 reconciliation/merge under normal policy, and then a fresh zero-PR scheduled/manual Hourly Product Development run.

Issue #22 therefore stays open only for criterion 5. No further local egress patch is justified by current runtime evidence.