Repository navigation
Native OpenMetrics /metrics endpoint with incremental collectors (replace polling prometheus-exporter) #1636
Description
Problem
Harper ships zero application metrics today — no harper*/hdb* custom metrics exist; the only app telemetry is the log stream (333GB/day, of which the entire extracted alerting signal is two restart-detection strings). Meanwhile hdb_analytics already collects rich series (utilization, table/database size, RocksDB stats, per-second raw analytics), and the external prometheus-exporter component is a known CPU hotspot because it polls heavyweight operations (effectively system_information) on every scrape (prometheus-exporter#44, #45).
Proposal
A core-native /metrics endpoint (operations API or built-in component) serving application/openmetrics-text (#376):
- Incremental in-memory collectors — counters/gauges updated at event time on each worker, aggregated across threads on scrape (same pattern as the status system's crossThread collector). No per-scrape system scans.
- Source existing
hdb_analyticscollectors where they're already cheap, plus the missing operational gauges:- ops/sec by operation type; per-resource request counts (Track and expose incoming request counts per resource #961)
- write/read transaction queue depth (Expose write (and read) transaction queue depth as a metric #592)
- cache hit rate / cache-hit duration (Add cache-hit-specific duration metric (contrast with cache_resolution) #682)
- replication per-peer: connection state, lag (applied vs head), resume-cursor position, blob in-flight (harper-pro#437 Tier 2)
- worker-thread availability ratio (get_status: include overall worker-thread availability ratio (for GTM load balancing) #668)
- restart counter + uptime (replaces the log-string restart monitors)
How it lands on the Grafana migration
This slots directly into the in-flight Datadog→Grafana migration (HarperFast/observability) as one of two complementary paths:
- Scrape path (fleet alerting + retention): the Grafana Alloy collectors already deployed on Fabric hosts scrape
/metricsinto Mimir — Harper app metrics land in the same stack as the existing 25 alert rules, so alert rules on replication lag or restart rate are just more Terraform in the observability repo. This path works when the instance is down/unreachable. - Pull path (deep drill-down, zero export cost): the published grafana-datasource plugin already queries
hdb_analyticscluster-wide (≥5.1.13 fan-out). High-cardinality diagnostic series (per-table, per-resource, per-peer detail) stay in Harper and are queried in place. New gauges surfaced throughhdb_analyticsappear here automatically.
Rule of thumb: low-cardinality health gauges → Mimir for alerting; everything else stays queryable in place.
This supersedes the polling architecture of prometheus-exporter (which can become a thin shim or be deprecated for v5+).
Related
#376, #592, #961, #682, #668, prometheus-exporter#44/#45, harper-pro#437 (Tier 2), HarperFast/observability, HarperFast/grafana-datasource.
🤖 Filed by Claude on behalf of Kris.
Activity
kriszyp commentedon Jul 16, 2026
From Chris Nelson's replication-monitoring feedback (2026-07-15) — two constraints to fold into the OpenMetrics endpoint design so the scraper side "just works":
1. Versioned status/metrics schema. ~36 older Fabric hosts are currently blind to the monitoring sidecar because of UDS/shape drift — the status shape changed under them and there was no negotiated version. The
/metrics(and underlying status) payload should carry an explicit schema version so scrapers can detect and tolerate skew instead of silently going dark. (No existing issue tracked this — capturing it here rather than as a standalone.)2. Unauthenticated scrape path. Chris underscored Prometheus export as his #1 ask, with the constraint that the scrape must work without unique per-host credentials for the Alloy collector — either exposed on the ops port (no auth) or via some other no-creds mechanism. Please make the auth model an explicit design decision here rather than defaulting to the app-port auth that would force per-host creds on every collector.
Ref: replication-side gaps also filed as HarperFast/harper-pro#589 (inbound-link awareness) and #590 (repl cert expiry) — both want per-peer labels on this endpoint.
🤖 Filed by KrAIs on behalf of Kris.
sleekmountaincat commentedon Jul 20, 2026
Scrape auth: acceptance criteria from the monitoring side
Following up on the unauthenticated-scrape constraint captured above, here is the precise requirement from the collector-fleet side, stated as acceptance criteria rather than a preference:
An Alloy collector must be able to scrape
/metricswithout unique-per-host credentials.Two shapes satisfy this; either is fine:
/metricson the operations port (or a dedicated metrics port) bound to loopback. This is the same trust model the current replication sidecar already uses (authless ops UDS, loopback listener), and it works on both cohorts today: fabric replication containers run host-network with loopback listeners, and legacy hosts run Alloy on-host beside the Harper container.metrics_scraperrole CM already provisions for the desired-state feed.What does not work: app-port auth with a distinct secret per host. That is 250+ unique secrets distributed to 250+ collectors, and it is the single design choice that would keep the shell+jq sidecar alive after this endpoint ships.
Also a +1 on the versioned-schema constraint riding with this: the ~36-host UDS shape-drift blindness is exactly the failure we want detectable at scrape time. A schema version in the payload lets collectors flag skew instead of silently going dark.
Rollout note so day one does not need to be perfect: we will run the existing sidecar in parallel per cohort and verify parity before deleting anything. The no-unique-creds requirement above is the only hard gate on our side.
🤖 Drafted with Claude Code on behalf of Chris Nelson