Skip to content

Fix live monitoring scrape coverage#288

Merged
Svaag merged 5 commits into
mainfrom
fix/live-monitoring-triage
Jun 21, 2026
Merged

Fix live monitoring scrape coverage#288
Svaag merged 5 commits into
mainfrom
fix/live-monitoring-triage

Conversation

@Svaag

@Svaag Svaag commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add the loop VM node_exporter target to the mon Prometheus scrape config so Prometheus-backed Icinga checks for loop resolve data
  • correct the internal Vault blackbox probe to use Vault's plain-HTTP internal listener and refresh the blackbox module comment
  • expose real Vector internal metrics on log via an internal_metrics source plus prometheus_exporter sink on :8686, and correct the Vector buffer alert metric name for Vector 0.55.0
  • disable netproxy Prometheus scrape targets because netproxy is intentionally not live yet; tracked in Complete live rollout of x402 Hyrule Network Proxy #214
  • refresh the generated log Vector artifact and network-flow docs

Live triage/remediation performed

  • copied current mon Prometheus/blackbox configs to mon and reloaded Prometheus/restarted blackbox exporter
  • applied the logs aggregator role on log to expose Vector Prometheus metrics on :8686
  • loop Prometheus-backed Icinga checks recovered: node-up, disk, engineering-loop-timer, knowledge-mcp-service are OK
  • Vector and internal Vault Prometheus probes recovered
  • disabled netproxy node_exporter/sidecar Prometheus targets live; active Prometheus targets now report no down targets
  • added triage notes to network-operations#214: Complete live rollout of x402 Hyrule Network Proxy #214 (comment)

Validation

  • scripts/ci/render-all.sh
  • scripts/ci/deploy-preflight.sh --repo-only
  • remote vector validate --config-toml on the rendered aggregator config
  • promtool check config /etc/prometheus/prometheus.yml on mon
  • Vector /metrics returns Prometheus samples including vector_buffer_byte_size
  • Prometheus active target check on mon: 55 active targets, 0 down active targets
  • Prometheus up == 0 returns empty

@Svaag Svaag requested a review from a team as a code owner June 21, 2026 19:34
@github-actions

Copy link
Copy Markdown
Contributor

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 1 🔵⚪⚪⚪⚪
🏅 Score: 90
🧪 No relevant tests
🔒 No security concerns identified
⚡ No major issues detected

@github-actions

Copy link
Copy Markdown
Contributor

PR Code Suggestions ✨

No code suggestions found for the PR.

@Svaag Svaag force-pushed the fix/live-monitoring-triage branch from f59760e to 294a768 Compare June 21, 2026 19:48

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 294a768a56

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ansible/roles/logs/templates/vector-aggregator.toml.j2 Outdated
@Svaag Svaag merged commit 153a705 into main Jun 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant