Prove rollback-by-SHA for hyrule-web and hyrule-cloud

[hyrule-engineering-loop]

Change class

app_feature

Repos touched

• hyrule-infra

Senior role reviews

• network_architect: not approved
• systems_engineer: approved
• devops_netops: approved
• security_auditor: not approved
• finops_integrity: not approved
• virtual_lab_chaos: not approved

Source-of-truth files consulted

• hyrule-infra:README.md

Validation gates run

• ['/opt/engineering-loop/.venv/bin/python3', '-c', 'import json\nfrom pathlib import Path\nfor raw in json.loads('["docs/runbooks/rollback-by-sha.md"]'):\n path = Path(raw)\n if not path.exists():\n continue\n if not path.is_file():\n raise SystemExit(f'not a file: {raw}')\n path.read_text(encoding='utf-8')\n']: passed

Expected production impact

none

Rollback plan

Discard the generated feature worktree and branch; no production state was changed.

NOC handoff

• handoff artifact: /var/lib/engineering-loop/runs/issue_network_operations_358/handoff/noc_handoff.json
• rollback trigger: operator rejection, failed gates, or failed post-deploy checks
• expected duration: none

Post-deploy checks

• review graph state
• run documented gates

Operator notes

Closes #358

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 358 - Partially compliant Compliant requirements: Document the runbook (docs/runbooks/rollback-by-sha.md): commands, verification checks, and the migration caveat. Non-compliant requirements: Pick a quiet window; record current good SHAs for hyrule-web and hyrule-cloud. Roll hyrule-web back one release by SHA via the promotion workflow; verify the site serves the prior version; roll forward again. Repeat for hyrule-cloud, including confirming Alembic migration compatibility posture for one-release rollback. Requires further human verification: The actual rollback execution steps (pick window, record SHAs, perform rollback, verify, roll forward) are not part of this PR and require human verification.

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪

🏅 Score: 95

🧪 No relevant tests

🔒 No security concerns identified

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Improve SHA verification method The verification step uses git rev-parse HEAD which may not reflect the deployed code if the repository is in a detached HEAD state or if the deployment uses a different mechanism. Consider using the application's version endpoint or a more reliable method to confirm the running SHA. docs/runbooks/rollback-by-sha.md [81-84] ssh -i ~/.ssh/id_servify root@2a0c:b641:b50:2::30 \ - 'cd /opt/hyrule-web && git rev-parse HEAD' + 'curl -sf http://127.0.0.1:8080/version 2>/dev/null || git -C /opt/hyrule-web rev-parse HEAD' # Must match the SHA you promoted. Suggestion importance[1-10]: 7 __ Why: The suggestion improves reliability by adding a fallback to a version endpoint, which is more robust than relying solely on git rev-parse HEAD. This is a moderate improvement for verification accuracy.	Medium
	Add log check for compatibility Running alembic current after a rollback may show the head revision from the newer code, but this does not guarantee N-1 compatibility. The runbook should explicitly check for application errors or warnings in logs after the rollback to confirm compatibility. docs/runbooks/rollback-by-sha.md [196-198] ssh -i ~/.ssh/id_servify root@2a0c:b641:b50:2::20 \ - 'cd /opt/hyrule-cloud && uv run alembic current' + 'cd /opt/hyrule-cloud && uv run alembic current && journalctl -u hyrule-cloud --since "5 minutes ago" --no-pager | grep -i "error\|warning" || true' Suggestion importance[1-10]: 7 __ Why: Adding a log check for errors/warnings after alembic current provides a more thorough validation of N-1 compatibility. This is a useful enhancement to the runbook.	Medium
	Include health check tags The emergency rollback section uses --tags apply which may skip pre-deployment checks or notifications. Consider adding a note to run the full playbook or include necessary tags to ensure all required steps (like health checks) are executed. docs/runbooks/rollback-by-sha.md [239-243] -ansible-playbook playbooks/web.yml --tags apply \ +ansible-playbook playbooks/web.yml --tags apply,health \ -e ansible_user=ci \ -e hyrule_web_apply=true \ -e hyrule_web_version="<PREVIOUS_KNOWN_GOOD_SHA>" \ --limit web Suggestion importance[1-10]: 6 __ Why: Including the health tag ensures that health checks are executed during emergency rollback, which is a good practice. The improvement is minor but adds safety.	Low