QA run · automation (FULL area) · a86db175 · 2026-08-11 · 5 PASS / 3 PARTIAL / 2 FAIL #7516

Full automation area run of the checklist-test skill — all 10 runnable items driven against a live showcase (2 opus subagents, isolated boot each). 1 item pre-blocked on fixtures (rollup-summary-filter). Labels: qa-run + bug.

Result: 5 PASS · 3 PARTIAL · 2 FAIL. Text-only per RUNNER.md. Both FAILs are checklist-vs-implementation contract disagreements with documented implementation-side rationale — neither is a regression. They need a maintainer decision, not a fix ticket.

Environment — framework a86db175 (PR #7304) · showcase · isolated file DB + port per batch · no outbound egress.

🟠 The two FAILs are contract decisions

1. flow-node-type-matrix — unknown node types are not refused at registration

POST /api/v1/automation with a node of type:'bogus_node' returns 200 and really registers the flow. It is not silent, which is the #1887 anti-goal: registration logs a WARN listing unknownTypes + knownTypes, and triggering answers success:false "No executor registered for node type bogus_node" with the run recorded failed. Implementation-side rationale is explicit — engine.ts registerFlow() / ADR-0018 §M1 / #4771: "membership is checked at that seam instead, and stays soft-fail — a flow authored against a currently-absent plugin must still register."

Worth an explicit ruling: the same endpoint hard-refuses an undeclared node config key (#4277, with a located message listing the declared keys) while accepting an unknown node type. One seam, two strictness levels.

Otherwise strong: 20 of 21 node types proven executed success from a 235-step sweep across ~100 runs; region tagging (loop-body/try/catch/parallel-branch + iteration + parentNodeId) all correct; every step names its nodeType (0 missing). Gap: end never appears as an executed step, only as a skipped branch target.

2. flow-error-handling — a failing try-region node produces no step at all

On a caught failure the steps are exactly [start, guarded_push(try_catch,success), record_failure(catch)] — nothing with regionKind:'try', nothing with status:'failure' — and the persistence layer agrees. Reproduced 2×. Deliberate per engine.ts runRegion() ("a failed attempt's partial steps are not surfaced") and try-catch-node.ts, which returns childSteps only from a successful region.

Operator consequence worth weighing: after a try_catch run you cannot tell what failed, how many attempts ran, or which node threw — only the catch's side effects.

Everything else in the item passes: $error interpolation into the record, the retry ladder (7.13s/7.16s matching 0+1+2+4s), unhandled failure terminating status=failed with both run-level and step-level errors, the designer Runs panel rendering region nesting, and the loud ERROR log for trigger-fired unheard failures. The "catch must not run when try succeeds" negative was proven with a purpose-built probe.

✅ PASS — 5 items

• durable-suspend-restart (5/5) — 6 paused runs byte-identical across a real cold restart on the same file DB; a PT1M wait timer elapsed on the new process with no manual resume and completed end-to-end (notification + receipt written); a nested subflow→approval pair both resumed from one /approve with the child's output bubbling to the parent.
• flow-toggle-kill-switch (4/4) — toggle OFF flips /_status enabled+bound to false (the trigger is genuinely unbound, not merely guarded) and a matching write produces no new run, asserted on bracketing reads.
• screen-flow-roundtrip (7/7) — and both negatives are the good kind: an empty-inputs resume is refused 400 naming the required field with the run left paused, and Cancel issues zero requests, leaves the run paused, and it remains resumable.
• flow-run-step-nesting (5/5) — the Studio panel renders a real indentation ladder (0/12/24px) with 1-based ITERATION headers, and the rendered tree reconciles exactly with the API step log.
• flow-runs-page-test-trigger (5/5) — Console: screen-flow Submit never calls the resume endpoint — every screen flow is un-completable from the UI #3528 refuted: a screen flow's paused envelope is handed to FlowRunner on the page, Submit resumes and the downstream write lands; dismissing instead leaves a working "Continue run" affordance, with the paused-row tally 0 before and after.

🟡 PARTIAL — 3 items

• trigger-type-matrix — all 6 trigger kinds fire and record their runtime kind; webhook HMAC intake 202 with payload interpolation, and both bad-signature and missing-signature refused 401 with no task created; anonymous /trigger denied 401 with no run row; schedule fired 12 consecutive runs exactly 60s apart. Two real gaps: trigger.recordId is never populated on record_change runs (so runs cannot be correlated back to the triggering record from the run log), and the persisted sys_automation_run history row carries no trigger block at all, so trigger kinds are lost across a process restart.
• time-relative-trigger — the sweep itself is solid (5 sweeps → exactly one run each, with the out-of-window and filter-excluded rows never firing, proven by the runs-per-sweep count rather than by inference; the both-windowing-modes build gate refuses at os lint and os compile with a located message and no artifact). Clause 0 blocked(fixture) — see below.
• connector-dispatch-matrix — MCP dispatch fully proven (run output upper=OBJECTSTACK); the unregistered-connector negative gives a named refusal, no silent no-op. Partial because the rest/declarative variants dispatch but capture nothing (run.output empty) — an authoring gap, not an engine one: those flows declare no isOutput variable while the MCP one does. Slack is blocked(environment).

Fixture / authoring issues to fix (they block clauses, or mislead)

1. Showcase REST connectors hard-wire baseUrl to http://127.0.0.1:3000 (StatusApi/StatusOpenApi providerConfig literals, not env-overridable), so self-ping flows fail fetch failed on any isolated instance — a port mismatch, not the sandbox egress block. Worked around with a throwaway TCP forwarder, giving clean before/after causal evidence. Needs an env-overridable baseUrl or a knownGaps entry.
2. Package-provided flows are read-only in Studio (no design mode, no node inspector for any node), so every clause needing a node panel on a shipped flow is unreachable. Affects time-relative-trigger c0 and connector-dispatch-matrix c3. A writable draft does open the inspector — verified, so this is a fixture limit, not a product defect.
3. A source comment overstates behaviour: TaskCompletedRestPingFlow / ShowcaseDeclarativeConnectorPingFlow claim "the call and its {status:'ok'} response are captured on the flow run" — they aren't, because neither declares an isOutput variable. Either add it (as the MCP flow has) or correct the comment and the checklist clause.
4. POST /api/v1/automation/<unknown>/toggle answers 500 INTERNAL_ERROR instead of 404 NOT_FOUND (P3 error-class mismatch).