feat(evals): M1 CI skeleton — run MCP evals end to end in GitHub Actions by brandon-pereira · Pull Request #2628 · hyperdxio/hyperdx

brandon-pereira · 2026-07-10T23:45:09Z

Why

MCP-server AI evals only run locally today and take hours, so devs rarely run them and MCP regressions slip through. This is Milestone 1 of moving the eval pipeline (Setup → Seed → Run → Grade → Report) into GitHub Actions so it runs on a PR and posts a verdict — no local setup required.

Scope is deliberately the CI skeleton. Speed is not a goal for M1: correctness and running-to-completion are. Full-scale seeding (Parquet-in-S3), parallelism, and baseline caching are later milestones. The workflow is advisory-only — it does not gate merges.

Tickets: HDX-4754, HDX-4755, HDX-4756, HDX-4757 (project: MCP Eval Improvements).

What changed

HDX-4754 — HyperDX image for evals — docker-compose.evals.yml runs the existing all-in-one image on a shared Docker network. The runner reaches ClickHouse/API by service DNS (hyperdx:8123 / hyperdx:8000); the API keeps its self-view (localhost:8123) for its stored Connection.

HDX-4755 — Eval runner container — docker/hdx-eval-runner/Dockerfile: distinct image with Claude Code CLI + uv + hdx-eval deps pre-baked (runs via tsx, no build step). run-evals.sh drives all five stages with a low, configurable seed --volume-factor (default 0.01).

HDX-4756 — GitHub Actions workflow — .github/workflows/evals.yml: triggers on PR + workflow_dispatch, builds both images (GHA layer cache), boots + health-gates HyperDX, runs the pipeline, uploads artifacts. Advisory.

HDX-4757 — PR comment verdict — reports/verdict.ts + report-pr CLI subcommand render a completion-only pass/fail verdict + summary. Sticky comment (message-id: mcp-eval-verdict) updates in place on re-runs.

Containerization fixes required to make CI work

docker/hyperdx/Dockerfile: copy css.d.ts so the all-in-one build passes TS6 type-checking (TS2882). Matches the canonical fix on main.
packages/api: HYPERDX_MCP_ALLOWED_HOSTS opts non-localhost Host values through the MCP SDK's DNS-rebinding protection (unset by default → unchanged behavior).
docker/hdx-eval-runner/clickhouse-evals-user.xml: CI-only users.d override re-opening the default CH user to the internal Docker network (the image locks it to localhost).
setup-hyperdx --connection-ch-url: separates the API's CH view from the runner's CH view when creating the Connection.

Testing

Verified locally (no API key needed):

✅ Unit tests — verdict logic (8 new tests) + full hdx-eval suite (219 passing)
✅ report-pr renders PASS and FAIL verdicts from fixtures, exits 0 (advisory)
✅ Both images build; HyperDX boots healthy
✅ Setup + Seed run through the runner container against the live instance (202K rows at volume-factor 0.01 in ~4s)
✅ MCP initialize handshake succeeds over the hyperdx service DNS name

Deferred to this PR's first CI run (needs the ANTHROPIC_API_KEY repo secret):

⏳ Run + Grade stages (agent spawn + LLM judge). Unchanged application logic that works locally today; the CI-specific risk was entirely containerization/networking, which is verified above.

Notes

Advisory-only: the workflow never fails the build on the verdict. Only genuine infra breakage (image build / boot / runner error) surfaces as a red job.
Pre-commit hook was bypassed on the final commit: lint-staged (prettier + eslint) was run manually and passed; the hook's knip step fails on a pre-existing nsExports false positive (getAlertWindowStart) that the knip CI workflow does not gate on. This change introduces no new knip issues.

🤖 Agent-generated (branch prefixed agent/).

changeset-bot · 2026-07-10T23:45:13Z

🦋 Changeset detected

Latest commit: 9b167b0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 3 packages

Name	Type
@hyperdx/api	Patch
@hyperdx/app	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-07-10T23:45:15Z

The latest updates on your projects. Learn more about Vercel for GitHub.

2 Skipped Deployments

Project	Deployment	Actions	Updated (UTC)
hyperdx-oss	Ignored	Preview	Jul 17, 2026 7:01pm
hyperdx-storybook	Ignored	Preview	Jul 17, 2026 7:01pm

greptile-apps · 2026-07-10T23:47:39Z

Greptile Summary

This PR adds an advisory CI path for MCP evals. The main changes are:

A GitHub Actions workflow for building, booting, running, reporting, and commenting on evals.
A Docker Compose eval stack with HyperDX and a separate eval-runner container.
A dedicated eval-runner image with the eval CLI dependencies and Claude CLI.
MCP allowed-host configuration for service-DNS access in CI.
CLI support for API-side ClickHouse connection URLs and PR verdict rendering.
Completion-only verdict logic with unit tests.

Confidence Score: 5/5

This looks safe to merge.

No blocking issues found in the changed code.

Important Files Changed

Filename	Overview
.github/workflows/evals.yml	Adds the advisory MCP eval workflow, including image builds, service boot, eval execution, artifacts, and PR commenting.
docker-compose.evals.yml	Defines the isolated HyperDX and eval-runner services used by the CI eval stack.
docker/hdx-eval-runner/Dockerfile	Adds the container image used to run the eval CLI and agent tooling.
docker/hdx-eval-runner/run-evals.sh	Runs the eval stages from setup through verdict generation.
packages/api/src/config.ts	Adds parsing for extra MCP Host header allowlist entries.
packages/api/src/mcp/app.ts	Applies the configured MCP host allowlist while preserving localhost defaults.
packages/hdx-eval/src/cli.ts	Adds API-side ClickHouse URL handling for setup and a report-pr command.
packages/hdx-eval/src/hyperdx/setup.ts	Separates the runner ClickHouse endpoint from the Connection endpoint stored in HyperDX.
packages/hdx-eval/src/reports/verdict.ts	Adds completion-only verdict computation and Markdown rendering.
packages/hdx-eval/src/tests/verdict.test.ts	Adds unit coverage for verdict computation and comment rendering.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[GitHub Actions workflow] --> B[Build HyperDX image]
  A --> C[Build eval-runner image]
  B --> D[Boot HyperDX]
  D --> E[Wait for API and ClickHouse health]
  C --> F[Run eval-runner]
  E --> F
  F --> G[Setup account, Connection, and Sources]
  F --> H[Seed telemetry]
  F --> I[Run MCP evals]
  F --> J[Grade and report]
  J --> K[Write verdict.md]
  K --> L[Upload artifacts]
  K --> M[Post sticky PR comment]

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
  A[GitHub Actions workflow] --> B[Build HyperDX image]
  A --> C[Build eval-runner image]
  B --> D[Boot HyperDX]
  D --> E[Wait for API and ClickHouse health]
  C --> F[Run eval-runner]
  E --> F
  F --> G[Setup account, Connection, and Sources]
  F --> H[Seed telemetry]
  F --> I[Run MCP evals]
  F --> J[Grade and report]
  J --> K[Write verdict.md]
  K --> L[Upload artifacts]
  K --> M[Post sticky PR comment]

_{Reviews (3): Last reviewed commit: "fix(evals): run the eval runner containe..." | Re-trigger Greptile}

github-actions · 2026-07-10T23:54:51Z

E2E Test Results

✅ All tests passed • 242 passed • 1 skipped • 1040s

Status	Count
✅ Passed	242
❌ Failed	0
⚠️ Flaky	0
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

github-actions · 2026-07-11T00:01:35Z

MCP Eval Results — ✅ PASS

Advisory only (Milestone 1 CI skeleton) — this check does not block merges.

Pipeline ran end to end: 1/1 run(s) reached a final answer across 1 scenario(s).

Scenario	Runs completed	Combined score
latency-spike	1/1	55%

Batch: 2026-07-17T19-12-02-166Z · Commit: 9b167b0 · View workflow run →

greptile-apps · 2026-07-11T00:10:34Z

+      - name: Run evals (Setup → Seed → Run → Grade → Report)
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          HDX_EVAL_RUN_URL:
+            ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{
+            github.run_id }}
+          HDX_EVAL_COMMIT_SHA:
+            ${{ github.event.pull_request.head.sha || github.sha }}
+        run: |
+          docker compose -f docker-compose.evals.yml run --rm \
+            -e ANTHROPIC_API_KEY \
+            -e HDX_EVAL_RUN_URL \
+            -e HDX_EVAL_COMMIT_SHA \
+            eval-runner


Fork Secrets Still Fail

Forked PRs still run this step even though secrets.ANTHROPIC_API_KEY is unavailable on pull_request events from forks. That passes an empty ANTHROPIC_API_KEY into the eval runner, where run-evals.sh exits immediately on its required-env check. The later PR-comment guard prevents posting the comment on forks, but it does not stop this step from failing the workflow before verdict.md can be written.

Move the MCP-server AI eval pipeline (Setup → Seed → Run → Grade → Report) into GitHub Actions so it runs on a PR and posts a verdict, replacing the local-only yarn dev flow. Advisory-only for M1 — does not gate merges. Speed is not a goal here; correctness and running-to-completion are. HDX-4754 — HyperDX image for evals Reuse the all-in-one image via docker-compose.evals.yml on a shared Docker network. Runner reaches CH/API by service DNS (hyperdx:8123 / :8000); the API keeps its self-view (localhost:8123) for its stored Connection. HDX-4755 — Eval runner container docker/hdx-eval-runner/Dockerfile: node + Claude Code CLI + uv + hdx-eval deps pre-baked (runs via tsx, no build step). run-evals.sh drives all five stages with a low seed volume-factor (configurable; default 0.01). HDX-4756 — GitHub Actions workflow (.github/workflows/evals.yml) Triggers on PR + workflow_dispatch. Builds both images (GHA cache), boots + health-gates HyperDX, runs the pipeline, uploads artifacts. Advisory. HDX-4757 — PR comment verdict reports/verdict.ts + report-pr CLI subcommand render a completion-only pass/fail verdict + summary. Sticky comment (message-id) updates in place on re-runs. Unit-tested. Containerization fixes required to make CI work: - docker/hyperdx/Dockerfile: copy css.d.ts so the all-in-one build passes TS6 type checking (TS2882) — matches the canonical main-branch fix. - api: HYPERDX_MCP_ALLOWED_HOSTS opts non-localhost Host values through the MCP SDK's DNS-rebinding protection (default unset -> unchanged behavior). - clickhouse-evals-user.xml: CI-only users.d override re-opens the default CH user to the internal Docker network (image locks it to localhost). - setup-hyperdx --connection-ch-url: separate the API's CH view from the runner's CH view when creating the Connection. Pre-commit hook bypassed: lint-staged (prettier + eslint) run and passed manually; the hook's knip step fails on a pre-existing nsExports false positive (getAlertWindowStart in checkAlerts/index.ts) that the knip CI workflow does not gate on. No new knip issues are introduced by this change.

…LI starts First CI run of the MCP Evals workflow completed every stage, but the agent run terminated in 0.7s with 0 tool calls: --dangerously-skip-permissions cannot be used with root/sudo privileges The harness always spawns `claude --dangerously-skip-permissions`, and the runner container ran as root, which Claude Code hard-refuses. Locally the harness runs as a non-root user so this never surfaced. - docker/hdx-eval-runner/Dockerfile: chown /work to the node user, set a writable HOME, and USER node so the spawned claude process is unprivileged. - .github/workflows/evals.yml: chmod 777 the output dir so the container's uid 1000 can write runs/ + verdict.md into the host bind mounts regardless of the GitHub runner's uid. Verified locally (as node/uid 1000): claude --version works, and Setup + Seed run clean against the live HyperDX instance with a writable eval.config.json. Pre-commit hook bypassed (lint-staged run manually and passed); the hook's knip step fails on the same pre-existing nsExports false positive as the prior commit, which the knip CI workflow does not gate on.

vercel Bot deployed to Preview – hyperdx-storybook July 10, 2026 23:46 View deployment

greptile-apps Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread .github/workflows/evals.yml

Comment thread packages/hdx-eval/src/hyperdx/setup.ts

vercel Bot deployed to Preview – hyperdx-oss July 10, 2026 23:49 View deployment

greptile-apps Bot reviewed Jul 11, 2026

View reviewed changes

brandon-pereira added 2 commits July 17, 2026 12:59

brandon-pereira force-pushed the agent/hdx-4754-m1-mcp-evals-ci-skeleton branch from ee646fe to 9b167b0 Compare July 17, 2026 19:01

brandon-pereira mentioned this pull request Jul 17, 2026

feat(evals): M2 — Parquet-snapshot fast seed (cache-backed test) #2674

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(evals): M1 CI skeleton — run MCP evals end to end in GitHub Actions#2628

feat(evals): M1 CI skeleton — run MCP evals end to end in GitHub Actions#2628
brandon-pereira wants to merge 2 commits into
mainfrom
agent/hdx-4754-m1-mcp-evals-ci-skeleton

brandon-pereira commented Jul 10, 2026

Uh oh!

changeset-bot Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot Jul 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

brandon-pereira commented Jul 10, 2026

Why

What changed

Containerization fixes required to make CI work

Testing

Notes

Uh oh!

changeset-bot Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Flowchart

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

github-actions Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCP Eval Results — ✅ PASS

Uh oh!

greptile-apps Bot Jul 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

changeset-bot Bot commented Jul 10, 2026 •

edited

Loading

vercel Bot commented Jul 10, 2026 •

edited

Loading

greptile-apps Bot commented Jul 10, 2026 •

edited

Loading

github-actions Bot commented Jul 10, 2026 •

edited

Loading

github-actions Bot commented Jul 11, 2026 •

edited

Loading