Route Discord alerts by operational domain

[Svaag]

Summary

• route network incidents to #noc, model/engineering-loop incidents to #ai, and workflow failures to the dedicated #ci secret
• make NOC Agent the single owner of normal Alertmanager/Icinga case cards, with a narrow six-hour direct fallback for loss of the NOC/monitoring plane
• remove legacy Gemini quota alerts, damp model-health flapping, and treat transient runtime/provider degradation as warning rather than model-chain outage
• keep extmon direct Discord only as an independent critical fallback while sending all extmon alerts through persistent NOC cases
• switch Alertmanager transport reassertion to 24 hours and enable acknowledgement-aware six-hour critical reminders in CaseService
• render route-specific webhook fields from Vault and document synthetic extmon validation

Dependencies

• Make Discord case alerts route-aware and durable noc-agent#72
• Suppress non-actionable engineering-loop Discord summaries engineering-loop#39
• live Discord cutover requires #ci plus Manage Channels/Manage Webhooks permission for the existing NOC Agent bot, or equivalent manual channel/webhook creation

Validation

• scripts/ci/render-all.sh
• scripts/ci/iac-static.sh — 122 tests passed; repository advisory host-tool warnings only
• focused routing and Vault contract tests passed

Rollout guard

Do not merge this draft before the app PR is promoted and route-specific webhook secrets are stored in Vault. Removing the normal direct Alertmanager receiver first would create an avoidable delivery gap.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🏅 Score: 92

🧪 PR contains tests

🔒 No security concerns identified

⚡ No major issues detected

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Check readiness and runtime for exit code The script collects readiness and runtime fields but does not use them to determine the exit code. According to the header comment, OK should only be returned when the model chain, runtime, and provider health are all OK; otherwise a WARNING should be raised. Include readiness and runtime in the condition to match the documented behaviour. configs/mon/icinga2/scripts/check_noc_agent_model_health.sh [67-75] case "$code" in 200) - if [ "$status" = "ok" ]; then + if [ "$status" = "ok" ] && [ "$readiness" = "ok" ] && [ "$runtime" = "ok" ]; then echo "OK - $detail"; exit 0 fi echo "WARNING - $detail"; exit 1 ;; 503) echo "CRITICAL - $detail"; exit 2 ;; *) echo "UNKNOWN - http=$code $detail"; exit 3 ;; esac Suggestion importance[1-10]: 8 __ Why: The suggestion correctly identifies that the script collects readiness and runtime fields but does not use them in the exit code logic, which contradicts the documented behavior in the header comments. This is a significant functional issue that could lead to incorrect alert states.	Medium
General	Strip whitespace from notification route env var Environment variable values from Icinga may contain trailing whitespace or newlines, causing the validation to silently fall back to the default "network" route. Strip the value before matching to ensure the intended route is preserved. configs/mon/icinga2/scripts/notify-noc-agent.sh [36-38] -notification_route = os.environ.get("NOTIFICATION_ROUTE") or os.environ.get("HOST_NOTIFICATION_ROUTE") or "network" +notification_route = (os.environ.get("NOTIFICATION_ROUTE") or os.environ.get("HOST_NOTIFICATION_ROUTE") or "network").strip() if notification_route not in {"network", "ai", "ci"}: notification_route = "network" Suggestion importance[1-10]: 6 __ Why: The suggestion addresses a potential robustness issue where trailing whitespace from Icinga environment variables could cause unintended fallback to the default route. While valid, the likelihood of this specific issue occurring in practice is moderate, and the impact is limited to misrouting rather than a critical failure.	Low

[Svaag]

Addressed both robustness suggestions in 4bb58ef:

• the Icinga model check now requires overall status, readiness, and runtime reliability to all be OK before returning exit 0;
• route values are stripped before being checked against the network/AI/CI allowlist.

The focused routing tests and the complete 123-test infrastructure suite pass.