[OPIK-8059] [QA] Proposed e2e specs: which rules score a playground dataset run, and rule filters

[CometActions]

Details

Two new Playwright specs proposed by the QA test-radar side flow, from an
exploratory pass over #7989 (OPIK-8059 fix: score playground dataset runs by
the selected rules). A human drove both flows by hand on the PR's own deployed
environment (pr-7989.dev.comet.com, build 2.2.41-7989-merge-3070) before
either was written; the exploration report is the source of the two candidates
below.

This was generated. It is a draft and needs review before merge.

Why it targets this PR's branch, not main

Base is jacques/OPIK-8059-playground-trigger-scope, deliberately.

Both specs assert behaviour that only exists on #7989: that the playground's
metric selector defaults to nothing ticked, that an unticked production-scope
rule no longer scores a dataset run, and that a rule's filters are skipped
entirely on the experiment path. On main today, shouldScoreTrace does not
make those distinctions, so both specs would fail there for the right reason.
They should land with the change they cover.

Written and run against the PR head, 1a086dff4b780965d7da2a856300542e8abbc07a.

The specs

1. e2e/tests/playground/playground-selected-metrics.spec.ts — @t2-cuj @area:playground

@cap:playground.run-against-dataset

Seeds three constant-score Python rules on one project (production/ticked,
production/unticked, experiment-scope/never ticked), runs a 3-item dataset
through the Playground UI, and asserts which rules scored it — both server-side
on the resulting experiment traces and in the score chips the output rows render.

Two tests:

• ticked subset — the selector opens at 0 of 3 selected (this is the flip:
  it used to default to all-ticked); ticking one rule moves it to
  1 of 3 selected; the run is then scored by exactly the ticked rule and the
  experiment-scoped one, and by nothing else.
• nothing ticked — the same seed left untouched is scored by the
  experiment-scoped rule only, while two enabled, full-rate production rules are
  skipped. This is the upgrade path: a legacy persisted null selection maps to
  "none ticked".

Both assert by equality on the whole score set per trace and on the total
chip count across the table, not by containment — the regression this exists
to catch is an extra rule scoring the run, which a toContain would not see.
The constant metrics return 1.0 for any input, so the only variable in the
result is which rules ran.

playground-smoke.spec.ts already covers this capability, but it stops at "an
experiment was created" — a run scored by the wrong set of rules ships green
under it.

2. e2e/tests/online-evaluation/online-evaluation-rule-filters.spec.ts — @t2-cuj @area:online-evaluation

@cap:online-evaluation.rule-filters (was covered: false)

API-level, no browser. One enabled rule at 100% and trigger_scope: both, with
a filter that cannot match. Its production (source=sdk) trace is skipped and
its experiment trace is scored 1.0 — filters are not consulted on the experiment
path at all.

The absence half is anchored on the rule's own log stream (… does not match the configured filters), not on a timeout, and the spec asserts it is the
filter skip rather than the neighbouring sampling-rate skip. It also gates on
reading the rule back before seeding anything: filters is absent from the
pinned SDK's evaluator shape, so a payload the backend quietly dropped would
leave an unfiltered rule behind and make the whole test pass for the wrong
reason.

Worth a reviewer's eye on the product, not the test: a rule filtered down to a
slice of production now scores every experiment trace in its project. The code
comment argues it ("an experiment run is not a stream") and
ignoresFiltersOnExperimentTraces pins it, so this spec encodes it as intended —
but it is a widening the PR title does not advertise.

Supporting changes

• core/metrics/ and the createAutomationRule / getAutomationRule /
  getAutomationRuleLogs / Location-header additions to
  core/backend/client.ts are back-ported verbatim from main, where they
  landed after this branch was cut. Copied rather than rewritten so the merge
  into main resolves as the same change. Two deliberate deviations, both
  needed here: createAutomationRule takes a filters list (filters is
  likewise absent from the pinned SDK's shape), and getAutomationRule returns
  it.
• createTraceWithSource moves to the raw REST write and gains endTime —
  also main's shape. OnlineScoringSampler drops every trace with a null
  end_time as a partial write, so a trace seeded without one is never scored,
  and a scoring spec built on it would assert nothing.
• pom/playground.page.ts gains metric-selector and score-chip locators.

Metric-selector test ids — added during review

MetricSelector originally carried no data-testid: its trigger is a bare
<div> (Radix PopoverTrigger asChild) and its rows are unlabelled <div>s, so
the POM addressed them through static text. That was flagged in review and is now
fixed in this PR — the rows already keyed on rule.id, so the handles were
cheap:

• playground-metric-selector-trigger on the trigger
• playground-metric-selector-row-${rule.id} on each rule row
• playground-metric-selector-summary on the " of selected" row

The POM addresses all three by test id. The original reason for deferring — the
verification environment served a prebuilt frontend image, so a spec depending on
a new testid could not have been run before being proposed — no longer applies:
the re-run below used a frontend built from this branch, so the test ids are
exercised rather than assumed.

Taxonomy

Updated in the same change: both specs added to their areas' specs: lists,
online-evaluation.rule-filters flipped to covered: true, tier: t2-cuj, and
playground.run-against-dataset's note extended to record what the new t2
coverage actually pins. rule-filters carries a note scoping the claim to the
API — no spec drives the filter editor in the rule dialog.

What I deliberately did not write

• The exploration's highest-priority finding is not in here.
  online-evaluation-sampling-rate.spec.ts's "A 0% rule skips every SDK trace
  but still scores experiment, playground and optimization traces" test (on
  main; it post-dates this branch) fails on this build, in three places:
  playground traces now go through the sampling rate, optimization traces are no
  longer scorable at all, and the the rate applies to production traces only
  log line it greps for is deleted. All three are intended consequences of this
  diff. That spec needs updating with this PR or it turns the merge red. It
  is an edit to an existing spec on an already-covered capability, so it does not
  belong in a new-test PR — flagging it here instead.
• Three other verified items from the exploration produced no spec: the
  free-form source=playground run and the full 8-rule × 5-source branch table
  are already pinned by backend unit tests (OnlineScoringSamplerTest) at a far
  lower cost than an e2e, and the score-cell polling observation is a timing
  measurement, not an assertion a spec can hold stable.

---

Testing

Both were run against the PR's own deployment (OPIK_BASE_URL=https://pr-7989.dev.comet.com,
OPIK_DEPLOYMENT=oss, workspace default) before this PR was opened.

cd tests_end_to_end/e2e
npx playwright test tests/online-evaluation/online-evaluation-rule-filters.spec.ts \
                    tests/playground/playground-selected-metrics.spec.ts \
                    --reporter=list --retries=0

  ✓  1 online-evaluation-rule-filters.spec.ts › A rule's filters gate its production traces and are ignored on its experiment traces (3.0s)
  ✓  2 playground-selected-metrics.spec.ts › A dataset run is scored by the ticked rule and the experiment-scoped rule, and not by the unticked one (20.6s)
  ✓  3 playground-selected-metrics.spec.ts › A dataset run with nothing ticked is still scored by the experiment-scoped rule, and only by it (22.3s)

  3 passed (31.4s)

All three passed, and passed again on a repeat run and at --workers=2. Because
pom/playground.page.ts and core/backend/client.ts are shared, the whole
tests/playground/ directory was re-run as well: 9 passed, no regressions.
(compare-json-key-sorting and optimization-trial-logs reach
createTraceWithSource on main; on this branch the new spec is its only
consumer.)

Also green: npx tsc --noEmit, and
python3 tests_end_to_end/coverage/tag_lint.py --taxonomy tests_end_to_end/coverage/taxonomy.yaml --estate tests_end_to_end
→ 0 problem(s).

Review re-verification (2026-08-27)

Reviewed via /review-generated-tests. Two changes landed on this branch as a
result, plus a merge:

1. Base merged in — the branch was 4 commits behind
   jacques/OPIK-8059-playground-trigger-scope, including
   fix(playground): address review findings on scoring and polling,
   perf(playground): do not poll a trace no rule will score and
   fix(online-scoring): stop scoring playground runs without a dataset, all of
   which touch the behaviour these specs pin. Merged clean, no conflicts.
2. Metric-selector test ids added and the POM switched onto them (see above).
3. The nothing-ticked test's comment corrected — it described itself as the
   legacy persisted-null upgrade path, but the spec never seeds
   scoresByDatasetId, so it exercises the fresh-session default. The claim now
   matches what it asserts; the store migration belongs in the
   metricSelection.ts unit tests.

Re-run against a stack built from this merged branch (backend + frontend from
source via scripts/dev-runner.sh, not a prebuilt image — the released images
predate #7989 and cannot exercise it):

tests/playground/ + tests/online-evaluation/   14 passed, 1 skipped (1.9m)

The skip is OpenRouter (Custom) in playground-providers — no
OPENROUTER_API_KEY present; pre-existing and unrelated. The full directories
were run, not just the new specs, because playground.page.ts is shared with 8
other specs: no regressions.

Gates on the merged tree: tag_lint 0 problems (47 specs) · tsc --noEmit clean
· playwright --list 3 tests in 2 files · eslint clean on the touched
component.

All four baz-reviewer findings are answered on their threads: two invalid (one
back-ported verbatim from main, one misreading conventions.md, which bans
deep-importing opik/rest_api/* rather than the suite's own rawFetch — already
the pattern on main at client.ts:985/:1018), two valid and fixed here.

One item for #7989, not this PR: main's online-evaluation-sampling-rate.spec.ts
greps 'the rate applies to production traces only' (lines 445, 481), a log line
this branch deletes from OnlineScoringSampler.java. That spec needs updating
with #7989 or the merge to main goes red.

Change checklist

• User facing
• Documentation update

Test-only, plus three data-testid attributes on MetricSelector that exist so
the specs can address it (no logic, styling or rendered-text change). The
"documentation" box covers the taxonomy note: updates, which are how this
estate records what a capability's coverage actually pins.

Issues

• OPIK-8059

Follow-up, not resolved here: main's online-evaluation-sampling-rate.spec.ts
greps a log line this branch deletes (OnlineScoringSampler.java:369), so it
needs updating with #7989 or the merge to main goes red. Tracked against that
PR rather than this one.

AI-WATERMARK

AI-WATERMARK: yes

• Tools: Claude Code — QA test radar (release-test-proposal / exploration side
  flow) authored the specs; /review-generated-tests reviewed them and applied
  the review fixes.
• Model(s): Claude Opus 5 (claude-opus-5)
• Scope: both spec files, the core/metrics/ + core/backend/client.ts +
  pom/playground.page.ts test support, the taxonomy entries, and the three
  MetricSelector test ids.
• Human verification: @andreic drove both flows by hand on pr-7989.dev.comet.com
  before either spec was written, and reviewed and directed the follow-up run
  described under Testing.

Documentation

No user-facing documentation change. Coverage bookkeeping lives in
tests_end_to_end/coverage/taxonomy.yaml and is updated in this PR: both specs
added to their areas' specs: lists, online-evaluation.rule-filters flipped to
covered: true, tier: t2-cuj with a note scoping the claim to the API, and
playground.run-against-dataset's note extended to record what the new t2
coverage pins.

[github-actions]

⏱️ pre-commit per-hook timing

Hook	Description	Result	Duration
🌐 typecheck — frontend	Whole-project tsc type check	✅	24.43s
🌐 eslint — frontend	Lint + autofix JS/TS	✅	3.13s
Total (2 ran)			27.56s

⏭️ 41 skipped (no matching files changed)

Hook	Description	Result
🐍 trim trailing whitespace — python sdk	Strip trailing whitespace	⏭️
🐍 fix end of files — python sdk	Ensure files end in a newline	⏭️
🐍 ruff — python sdk	Lint + autofix Python (ruff)	⏭️
🐍 ruff-format — python sdk	Format Python code (ruff)	⏭️
🐍 mypy — python sdk	Static type check	⏭️
🤖 trim trailing whitespace — optimizer	Strip trailing whitespace	⏭️
🤖 fix end of files — optimizer	Ensure files end in a newline	⏭️
🤖 check yaml — optimizer	Validate YAML syntax	⏭️
🤖 check json — optimizer	Validate JSON syntax	⏭️
🤖 check toml — optimizer	Validate TOML syntax	⏭️
🤖 check for added large files — optimizer	Block large files (>1MB)	⏭️
🔐 detect private key — optimizer	Block committed private keys	⏭️
🤖 check for merge conflicts — optimizer	Block merge-conflict markers	⏭️
🤖 check for case conflicts — optimizer	Block case-only name clashes	⏭️
🤖 pyupgrade — optimizer	Modernize Python syntax	⏭️
🤖 ruff — optimizer	Lint + autofix Python (ruff)	⏭️
🤖 ruff-format — optimizer	Format Python code (ruff)	⏭️
🤖 mypy — optimizer	Static type check	⏭️
📓 nbstripout — optimizer notebooks	Strip notebook output	⏭️
📝 markdownlint — optimizer	Lint Markdown	⏭️
🔤 codespell — optimizer	Fix common misspellings	⏭️
📊 radon cc — optimizer	Cyclomatic-complexity gate	⏭️
📊 radon raw — optimizer	Raw size metrics gate	⏭️
📊 xenon — optimizer	Fail on complexity thresholds	⏭️
📊 lizard — optimizer	Cyclomatic-complexity gate	⏭️
🧹 vulture — optimizer	Find dead code	⏭️
🛡️ trim trailing whitespace — guardrails	Strip trailing whitespace	⏭️
🛡️ fix end of files — guardrails	Ensure files end in a newline	⏭️
🛡️ ruff — guardrails	Lint + autofix Python (ruff)	⏭️
🛡️ ruff-format — guardrails	Format Python code (ruff)	⏭️
🛡️ mypy — guardrails	Static type check	⏭️
⚓ helm-docs	Regenerate Helm chart README	⏭️
block non-public FE plugins	Block non-public FE plugins	⏭️
☕ spotless — java backend	Format Java code	⏭️
🧪 pre-commit wrapper smoke tests	Self-test the wrapper scripts	⏭️
🧪 rebaseline script tests	Self-test the changelog re-baseline script	⏭️
📘 eslint — typescript sdk	Lint + autofix JS/TS	⏭️
📘 typecheck — typescript sdk	Whole-project tsc type check	⏭️
⚙️ actionlint — github workflows	Lint GitHub Actions workflows	⏭️
🐳 hadolint — dockerfiles	Lint Dockerfiles	⏭️
🌈 zizmor — github workflows security	Security-scan GitHub Actions workflows	⏭️

[AndreiCautisanu]

Generated-test review — /review-generated-tests

Verdict: ready. No blockers. Reviewed against the estate's own conventions
(qa-plugin@3f28e6e); the two should-fixes from the first pass are resolved and
the specs now have execution evidence on a stack that can actually run them.

What changed on the branch

Base merged in	was 4 commits behind, including fix(playground): address review findings on scoring and polling, perf(playground): do not poll a trace no rule will score and fix(online-scoring): stop scoring playground runs without a dataset — all touching what these specs pin. Clean merge, no conflicts.
Metric-selector test ids	added to MetricSelector.tsx (trigger, per-row by rule.id, summary) and the POM moved onto them.
Nothing-ticked comment	corrected — it claimed the persisted-null upgrade path, which the spec never seeds.

Execution evidence

The released images (opik-backend:2.2.35, opik-frontend:latest) predate
#7989, so they cannot exercise it — metricSelection.ts differs from main by
12/27 lines on this branch, and that diff is the "0 of N selected" assertion.
Backend and frontend were therefore built from the merged branch and the suite
run against that:

tests/playground/ + tests/online-evaluation/   14 passed, 1 skipped (1.9m)

Full directories, not just the new specs, since playground.page.ts is shared
with 8 others — no regressions. The one skip is OpenRouter (Custom), no key
present, pre-existing.

Gates: tag_lint 0 problems / 47 specs · tsc --noEmit clean · playwright --list 3 tests in 2 files · eslint clean on the touched component.

Bot findings

#	Finding	Verdict	Disposition
1	TraceJsonSection excludes boolean/null	invalid	byte-identical to origin/main:173 — back-ported, not authored here; no caller passes either type
2	createAutomationRule uses rawFetch against the convention	invalid	conventions.md bans deep-importing opik/rest_api/*, not the suite's own rawFetch; already the pattern on main (client.ts:985, :1018)
3	Brittle metric-selector handles	valid	fixed here — test ids added and consumed
4	Legacy-null migration untested	valid	fixed here — the overclaiming comment corrected; store migration belongs in metricSelection.ts unit tests

Each is answered on its own thread.

What holds up well

Tag honesty is the thing this review exists to check, and it is sound: both
@cap: keys exist, both fail if the capability breaks, and the assertions use
set equality plus a total chip count — an extra rule scoring the run fails,
which toContain would have missed. @t2-cuj is right per TESTING-TAGS.md
(LLM spend disqualifies t1). No fixed sleeps, no skips, literal tags. Cleanup is
in the shared automationRulesCleanup fixture with nothing in the test bodies,
and the PR adds no near-duplicate fixtures. The getAutomationRule read-back
before seeding is the sharpest part — filters is absent from the pinned SDK's
shape, so a silently-dropped payload would otherwise have made the test pass for
the wrong reason.

For #7989, not this PR

main's online-evaluation-sampling-rate.spec.ts greps
'the rate applies to production traces only' (lines 445, 481) — a log line this
branch deletes from OnlineScoringSampler.java:369. That spec needs updating
with #7989 or the merge to main goes red.

---

Marked ready for review at @andreic's explicit request. Note the standing rule
that this skill does not flip generated PRs itself — the draft flag is the human
gate — so this was a deliberate operator override, not the review clearing its
own gate.