Skip to content

Add experimental unified-session model routing analytics - #18475

Merged
pelikhan merged 9 commits into
mainfrom
pelikhan-model-routing-dashboard
Oct 17, 2026
Merged

pelikhan merged 9 commits into
mainfrom
pelikhan-model-routing-dashboard

Conversation

@pelikhan

@pelikhan pelikhan commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Why

Routing choices, response credits, and missing telemetry are difficult to investigate from aggregate workflow results. This adds a Model routing page and per-run drill-down based on the analytics specification in github/gh-aw#69507, under the Experimental navigation section.

Approach

  • Project metadata-only routing evidence from usage/aw_session.jsonl into the existing canonical Audit collection and typed PostgreSQL columns, without creating a parallel telemetry source. Prompts, bodies, free-form errors/reasons, and content-derived hashes are excluded.
  • Declare routing state precedence, run-level filters, adoption, classifier dimensions, pick stability, response attribution, complete-cost distributions, observational comparisons, deviations, and agent outcomes in Dashboard Language. Extend shared quantile/window operations and parameter resolution consistently across the worker and hosted query engine.
  • Add responsive run navigation, accounting/collection coverage, and timelines. Unknown costs remain unknown; partial costs do not enter complete-cost comparisons. Exact and correlated sub-agent attribution remain distinct.

Scope and deployment notes

This is a substantial implementation, not full conformance with every upstream metric. Candidate ranking, inventory/router change context, alias moves, prioritized deviation causes, failure clustering, and full safe-output/grader/delayed-outcome comparisons remain documented gaps.

Canonical schema 31 / IndexedDB version 41 rebuild older browser projections. Hosted PostgreSQL requires a refreshed generated schema. Generated aggregate query catalogs now share the existing aggregate-dashboard size-check exception; authored fragments remain size-checked.

Validation

After merging main through 8332050ca:

  • npm run check passed, including root tests, schema checks, workflow compilation, and documentation build.
  • Full dashboard unit suite: 3,466 passed, one skipped. Dashboard lint, typecheck, and document validation passed.
  • Full browser suite: 270 passed, one skipped, including desktop/mobile routing, reactive filters, Experimental navigation, drill-down, and the upstream ingestion fixes. The previously failing ingestion smoke assertion now passes.
  • Native PostgreSQL query tests, collection tests, and Go vet passed. Earlier PostgreSQL storage tests and the isolated deployed-data worker-memory budget run also passed.
  • Populated desktop/mobile screenshots are attached in a comment.

Outstanding performance check: the earlier Lighthouse run reported CLS 0.137 against the 0.1 budget in populated CFO mobile scenarios; other measured scenarios passed. Lighthouse was not rerun after the base merge, so this is not reported as green.

pelikhan and others added 5 commits October 16, 2026 18:36
Project metadata-only routing evidence into canonical Audit storage and implement declarative adoption, attribution, cost, stability, deviation and outcome views with run drill-down.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…ents

Retain the size limit on authored fragments, matching the existing aggregate dashboard document exception.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@pelikhan

Copy link
Copy Markdown
Collaborator Author

Populated Model routing page under Experimental, at desktop and mobile widths. These are full-site screenshots rendered through the SQLite worker/query boundary from unified-session fixture evidence, not mocked page payloads.

Model routing desktop at 1280 pixels

Model routing mobile at 390 pixels

@pelikhan
pelikhan marked this pull request as ready for review October 17, 2026 01:48
Preserve routing and shard-date compute functions, regenerate the combined query catalog, and keep Model routing in the Experimental navigation contract.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor
Dashboard dead views

Dead element views (0):
Dead dashboard pages (0):
Analyzed 8 element renderers and 65 dashboard pages across 1 dashboard.json file(s) from dashboard/site/src/components/ui-elements.js.

⚠️ This reachability report is informational and does not block the pull request.
🔎 View the workflow run and report artifact.

@pelikhan

Copy link
Copy Markdown
Collaborator Author

@copilot merge main and recompile

Copilot AI and others added 3 commits October 17, 2026 05:49
…g-dashboard

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>
Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI commented Oct 17, 2026

Copy link
Copy Markdown
Contributor

@copilot merge main and recompile

Merged main at 4df4baeee in b7f87576b and recompiled all 72 workflows; no lock-file changes. Follow-up fixes through fb771ec66 address Go lint and preserve numeric routing status/HTTP evidence.

npm run check, 309 focused dashboard tests, dashboard lint/typecheck, and Go lint/query/Postgres tests passed. Full dashboard suite was stopped after timeouts; affected tests passed in isolation. Worker-memory validation was blocked by deployed-site DNS resolution. Go CodeQL found no alerts; remaining language analysis timed out.

@pelikhan
pelikhan merged commit 67769c8 into main Oct 17, 2026
32 checks passed
@pelikhan
pelikhan deleted the pelikhan-model-routing-dashboard branch October 17, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants