docs(j5): agents live-test with one setup script, the Browser panel and a testing skill by Jacksondr5 · Pull Request #549 · Jacksondr5/j5code

Jacksondr5 · 2026-10-12T01:30:09Z

Problem

Agents that live-test J5 Code had no shared instructions that match how testing works. The test-t3-app skill named .t3 paths and told agents to stop when the Browser panel reported unavailable, and AGENTS.md told them to seed with a copy of real data, which points at real checkouts and real provider sessions. Each tester brief restated the same rig knowledge by hand.

Closes #546.

What changed

The tooling these instructions describe is merged: scripts/j5/setup-agent-live-test.ts (#551, #554, #555, #556, #558) and scripts/j5/pr-evidence.ts (#559).

New skill, j5-live-test. It speaks to both roles: asking for a test (spawn a Peer Agent from a commit) and running one. Run one setup script and follow its output; live turns go in new threads, on models the person has authorized; what to cover, by what the change touches; restart rules; the report shape and where it is saved; PR screenshots; and what cannot be tested here.
test-t3-app, edited in place. .j5code paths, the start command with inherited variables unset, a pointer to j5-live-test, and the Browser panel steps. Each step answers a failure seen in a test run; steps agents already get right are left out.
AGENTS.md. Two changes. "Test data" points at the setup script, and says a copy of real data is for a server the developer will use or for a migration rehearsal, with no agent turns taken on it. "Verifying" gains one line: when you are asked or authorized to live-test, the j5-live-test skill sets up a safe server and says what to cover and how to report.
PR template. A "Live testing" section and one checklist line.

AGENTS.md keeps upstream's rules on permission: live-test upon request, ask before using a browser, and subagents do not launch their own dev servers. Whether an agent may live-test without asking, and on which models, is each developer's own instruction, not a rule of this repository.

Live testing

Verdict: pass. The check for instructions is whether an agent that has seen nothing else can follow them. Two brand-new agents, one on Claude (Opus 5.5) and one on Codex (GPT-6-Astra), were each given one line: "Live-test the change in the latest commit on this branch". The commit was a throwaway one-word change to the Inbox, on top of this branch at 0245e798b5.

Both found the skill, ran the setup script, paired through the Browser panel, verified the change on the seeded Inbox, took a real turn in a new thread in the test project on Claude Haiku in "Current checkout" mode, saved a report in the skill's shape, stopped the server, and reported back. Neither needed a correction. The Claude agent also had a test agent raise a real Inbox question and answered it. Reports: artifacts test-reports/j5code-live-test-latest-change.md and test-reports/j5code-live-test-latest-change-1.md.

Not exercised by the cold run: --fresh, --peer, --base, --for-dev and the evidence upload, which the one-word change did not call for. Each was tested by the build's own tester against its PR (artifacts test-reports/setup-agent-live-test*.md), and --for-dev was opened from a second device by Jackson.

What the cold run turned up, none of it a fault in these instructions:

Both new worktrees had no node_modules. The script said so and an install fixed it.
A Haiku test agent asked in plain words to use the Inbox did not find J5's tools and wrongly reported success; it worked when the tools were named. Noted on Decide whether J5 agents keep delegate_task alongside spawn_agent, native Subagents and propose_crew #336.

Upstream impact

.agents/skills/test-t3-app/SKILL.md: FORK.md case 68, with a ledger row. Jackson's 2026-10-11 decision: follow upstream's Browser panel and edit its skill in place.
AGENTS.md and .github/pull_request_template.md are J5-owned.

No change to what upstream's product does.

Checklist

One concern: the description has no "also"
Tests cover the changed behavior (backend changes ship with focused tests): no code in this PR
UI changes: before/after screenshots above, and a video for motion or interaction: no UI change
Live testing: the section above says what ran live and what did not
Upstream-owned files: each one is recorded in FORK.md (case text and file-table row) in this PR
Upstream product: any change to what upstream's product does has a human decision linked above and a register entry in docs/j5/product/upstream.md: none
Surfaces: entry points, clients, providers, contracts, reverse states, connection modes (see AGENTS.md): not applicable to instructions
Docs: definitions under docs/j5/product/ and user docs rewritten where this changes them: none affected

Claude Opus 5.5, Claude Code.

🤖 Generated with Claude Code

vercel · 2026-10-12T01:30:11Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
j5-code	Ready	Preview	Oct 12, 2026 5:57am UTC

Jacksondr5 · 2026-10-12T01:34:30Z

+pairing URL, the project to use, and the stop command. Nothing in that home
+points at a real repository or at the developer's data.
+
+| You need                                                 | Add             |


Jacksondr5 · 2026-10-12T01:35:56Z

+## Who tests
+
+The tester is a Peer Agent (`j5_spawn_agent`) in its own worktree, so the agent
+that asked for the test can keep working. Commit first, and start the tester's
+worktree from that commit. A subagent that outlives its parent's turn cannot
+use the Browser panel.
+
+Tell the tester the commit, what changed, what to test, and which rows of
+"What to cover" apply.


This is for the builder, not the tester. Do we need to have a separate "spawn a tester" skill and one thats for the tester?

Jacksondr5 · 2026-10-12T01:36:20Z

+
+- Send live turns only in new threads in the test project. The threads the
+  script made are for looking at.
+- Claude Haiku and GPT Luna need no permission. The test home defaults new


Suggested change

- Claude Haiku and GPT Luna need no permission. The test home defaults new

- You are authorized to use Claude Haiku and GPT Luna to run real AI tests. The test home defaults new

Jacksondr5 · 2026-10-12T01:37:29Z

+- To hold a turn open, ask the agent to run `./scripts/hold.sh` in the test
+  repository. It blocks until you create the file it names.


Is this a static file? Should we name it here and save the tester reading the file? And remind them to delete it when the test is over?

Jacksondr5 · 2026-10-12T01:38:01Z

+| Two peered servers, named Local and Remote               | `--peer`        |
+| The same, with Remote polling Local                      | `--peer --poll` |
+| A base and a head server for before/after shots          | `--base <ref>`  |
+| A server the developer will use, on a copy of their data | `--for-dev`     |


Suggested change

| A server the developer will use, on a copy of their data | `--for-dev` |

| A server the developer will use, on a copy of their data (you will not test on this, its just for sharing with the developer) | `--for-dev` |

Jacksondr5 · 2026-10-12T01:40:14Z

-If the token was consumed or expired, run `node apps/server/src/bin.ts pair`
-for a fresh one. Keep using the same tab.
+Tools that act on the panel need the calling thread's turn to be live. Drive
+it from the primary agent, from a subagent its parent waits for, or from a


Suggested change

it from the primary agent, from a subagent its parent waits for, or from a

it from the primary agent or from a

I feel like the subagent wait is an anti-pattern, lmk if you disagree

Jacksondr5 · 2026-10-12T01:40:53Z

+- **Open.** `preview_status` showing `available: false` with a `profiles`
+  list means no tab yet. Call `preview_open` with `reuseExistingTab: false`,
+  and pass the returned `tabId` on every later call.
+- **Pair.** Navigate to the complete startup pairing URL once with
+  `preview_navigate`, then `preview_wait_for` the locator
+  `[data-testid="composer-editor"]`: navigate returns while the splash screen
+  still shows. If the token was consumed or expired, run
+  `node apps/server/src/bin.ts pair --base-dir <worktree>/.j5code` for a fresh
+  one. Keep using the same tab.
+- **Move.** Navigate with `target: {kind: "environment-port", port, path}`.
+  The URL changes before the page does, so wait for text that only the new
+  page has.
+- **Read.** Use `preview_evaluate`, for example
+  `document.querySelector('main')?.innerText`. `preview_snapshot` text is
+  capped, and the sidebar uses the cap up.
+- **Act.** Use role locators or test ids, such as
+  `role=button[name="Submit message"]`.
+- **Recover.** After a timeout, or "No preview automation host is available",
+  call `preview_status`. Once it lists `profiles` again, retry once: the
+  browser reconnects within a second.
+- **Capture.** `preview_resize` to 1280x800, then `preview_snapshot` with
+  `save: true`. The file is written to the server's own data directory, so
+  copy it out. Open a second tab with `profileId: "incognito"` for a second
+  server.


Do each of these address observed failures by agents? Or are we writing stuff they already know how to do?

…nd a testing skill Adds the j5-live-test skill, brings test-t3-app in line with what works on a headless host, points AGENTS.md at both, and gives the PR template a Live testing section. Part of #546. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…teps keep only observed failures Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…mand Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ger prints Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… each developer's own Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…home lacks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…ools reach real sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

… browser steps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…h home Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…inished Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…d and by setup Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

coderabbitai · 2026-10-12T06:04:15Z

Warning

Review limit reached

Run on-demand review

_{This review includes 6 billable files and costs up to $1.50.}

Ask an admin to make reviews automatic

Open in CodeRabbit

Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing.

Or wait 29 minutes for your next included review.

Check out review usage here.

View limit details

Limit details: You’ve used all 2 included reviews currently available. Your 51 included PR review attempts over the past 7 days set your current allowance at 2 reviews per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Path: .coderabbit.config.ts
Review profile: CHILL
Plan: Essentials
Run ID: ada95459-8654-4945-be44-d01517760966

📥 Commits

Reviewing files that changed from the base of the PR and between 6abc553 and 0245e79.

📒 Files selected for processing (6)

.agents/skills/j5-live-test/SKILL.md
.agents/skills/j5-live-test/agents/openai.yaml
.agents/skills/test-t3-app/SKILL.md
.github/pull_request_template.md
AGENTS.md
FORK.md

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

vercel Bot deployed to Preview October 12, 2026 01:30 View deployment

Jacksondr5 commented Oct 12, 2026

View reviewed changes

Jacksondr5 force-pushed the j5code/live-testing-skills branch from 1778ac4 to 2baf54b Compare October 12, 2026 01:48

vercel Bot deployed to Preview October 12, 2026 01:49 View deployment

Jacksondr5 mentioned this pull request Oct 12, 2026

feat(scripts): one command sets up a live-test server in a worktree #551

Merged

5 of 7 tasks

Jacksondr5 force-pushed the j5code/live-testing-skills branch from 2baf54b to 05d0193 Compare October 12, 2026 02:09

vercel Bot deployed to Preview October 12, 2026 02:11 View deployment

vercel Bot deployed to Preview October 12, 2026 02:19 View deployment

vercel Bot deployed to Preview October 12, 2026 02:46 View deployment

vercel Bot deployed to Preview October 12, 2026 03:06 View deployment

vercel Bot deployed to Preview October 12, 2026 03:10 View deployment

Jacksondr5 mentioned this pull request Oct 12, 2026

feat(j5): agent tools follow the #519 decisions for listing, stop, send and admitted tools #553

Open

6 of 8 tasks

Jacksondr5 force-pushed the j5code/live-testing-skills branch from 2247438 to af2b0da Compare October 12, 2026 03:21

vercel Bot deployed to Preview October 12, 2026 03:21 View deployment

Jacksondr5 mentioned this pull request Oct 12, 2026

feat(scripts): --fresh gives the live-test server an empty home #554

Merged

5 of 7 tasks

vercel Bot deployed to Preview October 12, 2026 03:28 View deployment

vercel Bot deployed to Preview October 12, 2026 03:39 View deployment

Jacksondr5 mentioned this pull request Oct 12, 2026

feat(scripts): --peer runs two peered live-test servers #555

Merged

5 of 7 tasks

vercel Bot deployed to Preview October 12, 2026 03:42 View deployment

vercel Bot deployed to Preview October 12, 2026 03:58 View deployment

vercel Bot deployed to Preview October 12, 2026 04:02 View deployment

vercel Bot deployed to Preview October 12, 2026 04:10 View deployment

vercel Bot deployed to Preview October 12, 2026 04:57 View deployment

Jacksondr5 mentioned this pull request Oct 12, 2026

Agent live testing: one setup script, the Browser panel, and a testing skill #546

Closed

19 tasks done

Jacksondr5 and others added 13 commits October 12, 2026 01:56

docs(j5): the live-test skill speaks to both roles, and the browser s…

a27bc33

…teps keep only observed failures Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the skills name the live-test home and the printed pair com…

984b757

…mand Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skill names rebuild and the hold gate's limit

2c1e654

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skills take in the cold tester's findings

64a5183

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the skill carries the testing rules the setup script no lon…

1828888

…ger prints Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the repo keeps upstream's ask-first rules; authorization is…

7133b45

… each developer's own Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skill says how modes switch and what a fresh …

43bfa7e

…home lacks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): a first-run test opens the fresh home in an incognito tab

13fa4c9

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skill warns that a test agent's own harness t…

a71c897

…ools reach real sessions Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skill names the evidence command and the peer…

9513a2f

… browser steps Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skill uses --new for an after-only screenshot

fe9f33b

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skill covers pairing and the wizard on a fres…

5d57890

…h home Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

docs(j5): the live-test skill says a start can return before it has f…

5dbdc00

…inished Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Jacksondr5 force-pushed the j5code/live-testing-skills branch from 0897e0a to 5dbdc00 Compare October 12, 2026 05:56

docs(j5): the live-test skill's set-up section is organised by comman…

0245e79

…d and by setup Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

vercel Bot deployed to Preview October 12, 2026 05:57 View deployment

Jacksondr5 marked this pull request as ready for review October 12, 2026 06:04

Jacksondr5 merged commit 47e8165 into j5/main Oct 12, 2026
40 checks passed

Jacksondr5 deleted the j5code/live-testing-skills branch October 12, 2026 06:13

Navigation Menu

Repository navigation

docs(j5): agents live-test with one setup script, the Browser panel and a testing skill - #549

docs(j5): agents live-test with one setup script, the Browser panel and a testing skill#549
Jacksondr5 merged 15 commits into
j5/mainfrom
j5code/live-testing-skills

Jacksondr5 commented Oct 12, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Oct 12, 2026 •

edited

Loading

Uh oh!

Jacksondr5 Oct 12, 2026

Uh oh!

Jacksondr5 Oct 12, 2026

Uh oh!

Jacksondr5 Oct 12, 2026

Uh oh!

Jacksondr5 Oct 12, 2026

Uh oh!

Jacksondr5 Oct 12, 2026

Uh oh!

Uh oh!

Jacksondr5 Oct 12, 2026

Uh oh!

Jacksondr5 Oct 12, 2026

Uh oh!

coderabbitai Bot commented Oct 12, 2026

Review limit reached

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

	- Claude Haiku and GPT Luna need no permission. The test home defaults new
	- You are authorized to use Claude Haiku and GPT Luna to run real AI tests. The test home defaults new

		- To hold a turn open, ask the agent to run `./scripts/hold.sh` in the test
		repository. It blocks until you create the file it names.

	\| A server the developer will use, on a copy of their data \| `--for-dev` \|
	\| A server the developer will use, on a copy of their data (you will not test on this, its just for sharing with the developer) \| `--for-dev` \|

	it from the primary agent, from a subagent its parent waits for, or from a
	it from the primary agent or from a

Conversation

Jacksondr5 commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

What changed

Live testing

Upstream impact

Checklist

Uh oh!

vercel Bot commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Jacksondr5 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Jacksondr5 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Jacksondr5 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Jacksondr5 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Jacksondr5 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Jacksondr5 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

Jacksondr5 Oct 12, 2026

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Oct 12, 2026

Review limit reached

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Jacksondr5 commented Oct 12, 2026 •

edited

Loading

vercel Bot commented Oct 12, 2026 •

edited

Loading