Server & build telemetry — operational dashboard for runner health, build performance & cost

[mmcky]

Summary

Sibling to #321. That initiative tracks how each lecture series is configured (declarative, point-in-time, stored in-repo as JSON + Pages). This issue covers the other half: operational telemetry — how our builds and runners actually perform, behave, and cost over time. Different data shape (time-series, not config), different cadence (continuous, not on-change), different audience (ops, not authors), and — unlike #321 — it genuinely needs a managed time-series backend rather than a GitHub-native store. We keep the two planes separate, joined only by a one-way link (a series page in the status dashboard can link out to its telemetry board).

Relationship to #321 (the two-plane split)

	Status & config (#321)	Server/build telemetry (this issue)
Data	declarative config (versions, pins, flags)	time-series metrics (durations, CPU, $)
Cadence	on migration / daily scrape	continuous
Storage	JSON-in-repo + Pages	time-series DB (CloudWatch / Grafana)
Retention	forever (git history = migration record)	rolling (14–30d)
Audience	maintainers, authors	ops

Overlap is just build duration: telemetry owns the trend; #321 keeps at most a labelled snapshot value.

Goal

A single operational view ("is the fleet healthy, are builds fast, what are we spending?") that can pull from a range of services rather than being wired to one provider.

Services / data sources to support

Source	Provides	Status
RunsOn (EC2, v3 Flex)	runner host metrics, control-plane health, cost — native OTLP + CloudWatch	primary
AWS / CloudWatch	EC2 instance + GPU metrics, AWS Budgets	primary
GitHub Actions API	workflow durations, success/failure, queue time — across all repos	v1/v2
GitHub-hosted runners (ubuntu-latest)	builds outside RunsOn (e.g. scipy_tutorial_2026)	via Actions API
Future (Pages/Netlify deploys, etc.)	deploy timing/success	extensible

What should be reported through this interface

A. Build / workflow — total + per-job duration (ci / cache / publish / collab), success & failure rate, failure classification (build error vs runner/infra vs cache miss vs dependency resolution), queue/wait time (trigger→pickup), re-run frequency, duration trend per series (regression detection after dep bumps).

B. Runner / host (RunsOn / EC2) — CPU, memory (peak + sustained), disk usage & IO (ties into the volume=80gb work), network IO, GPU utilization + GPU memory (g4dn — are we paying for GPU we don't use?), boot time, spot vs on-demand mix, spot interruptions, instance type / AMI in use.

C. Cost — $/build, $/day, $/repo, $/series, spot vs on-demand split, GPU vs CPU runner cost, idle/waste (instances up but not executing), month-to-date vs AWS budget.

D. Capacity & control-plane health (RunsOn Flex) — queue depth / backlog, unschedulable jobs, GitHub API rate-limit pressure, webhook latency / Lambda errors, RunsOn license status.

E. Alerting — unschedulable jobs, failure-rate spikes, cost/budget anomalies, stuck-runner / hard-limit timeouts → SNS → Slack/email.

Candidate tooling (no self-hosted webapp)

1. Start with RunsOn's native v3 Flex surfaces — CloudWatch stack dashboard + daily cost reports + inline per-job charts, zero new infra. Measure the gap first.
2. Unified pane: Grafana Cloud (free tier) fed by RunsOn OTLP (extras=otel + OtelExporterEndpoint) for the runner/EC2 side + Grafana GitHub data-source plugin for the cross-repo workflow layer.
3. Alternative: Amazon Managed Grafana ← CloudWatch to stay entirely inside AWS.

Open questions

• Backend: native RunsOn only, Grafana Cloud (OTLP), or Amazon Managed Grafana (CloudWatch)? Decide after evaluating native.
• v1 scope: RunsOn/EC2 only, or include the cross-repo GitHub Actions workflow layer from the start?
• Retention window (14 / 30 / 90 days)?
• GPU utilization via OTLP host metrics vs runs-on/action@v2 CloudWatch (bills per metric)?
• Alert routing + ownership (Slack? email? who's on point?).

Suggested first step

Confirm/enable RunsOn v3 Flex's native CloudWatch dashboard + daily cost reports, capture what's missing, then choose a backend.

[mmcky]

Case study (2026-08-04): nine months of silent scheduled-workflow failure

While working the lecture-python-programming tracker audit we found the three "Execution Tests [Latest Anaconda]" workflows (execution-linux.yml daily, execution-osx.yml / execution-win.yml weekly) had not produced a green run since 2025-11-02 — 353 consecutive failed scheduled runs (274 linux / 40 osx / 39 win) by the time the fix landed. The tracker only noticed on 2026-07-08 (QuantEcon/lecture-python-programming#569), eight months in, and even then only the latest failure mode was visible: jupyter-book 2.x becoming the default pip target turned an unpinned install into jb: command not found. Whatever was failing between 2025-11 and 2026-05 is now undiagnosable — GitHub's ~90-day log retention means only pass/fail conclusions survive that far back, not causes. The eventual fix was a one-line pin (QuantEcon/lecture-python-programming#596), validated green in six minutes.

What made this invisible maps directly onto this issue's sections:

Gap	Section above
Scheduled runs annotate no PR and page no one, so 353 red runs never crossed a human's screen	A — success & failure rate
The failure class changed mid-outage (unknown → dependency resolution) and nothing surfaced the transition	A — failure classification
No alert on "scheduled workflow has been red for N days"	E — alerting
Post-hoc diagnosis impossible past 90 days; even coarse failure classification stored as time-series would have outlived the logs	A + retention
The workflows were schedule-only (no workflow_dispatch), so they couldn't even be manually probed without first editing the file	operability (worth a line in B/D)

Two takeaways for the open questions:

1. The cross-repo GitHub Actions layer earns its place in v1. This outage ran entirely on free GitHub-hosted runners — RunsOn/CloudWatch telemetry would never have seen it. The cheapest metric in the whole proposal, days since last green run, per scheduled workflow, per repo (straight from the Actions API), would have flagged this at day 2 instead of month 9.

2. Humans anchor on the visible failure mode. Even a deliberate tracker audit estimated the outage at ~9–10 weeks, because that is how long the current error message had been appearing; only last-green timestamps exposed the real window. Trend data needs to live in the telemetry plane, not in issue threads.

🤖 Generated with Claude Code