Teach a bot a routine by showing it the task #452

Problem

Users can only teach a bot a repeatable task by writing the routine out themselves. A user should be able to show the bot the task once, review what it observed, and save it as a routine, without the bot turning an observation into a routine on its own.

On main, routines already have drafts, approval, dry runs, scheduling, pause and resume, and run history (Routine*Command contracts, RoutineRuntime, the shared routine state, RoutineFormDialog, and RoutineRunHistory). There is no teaching-by-demonstration pipeline. apps/web/src/browser/browserRecording.ts records preview video, which cannot become a reusable routine on its own.

Goal

A user shows a bot one task, reviews the observed steps, chooses among clear options, edits the proposed procedure, and saves it only after approving it.

• Before saving, the review shows the sites, tools, connectors, write actions, and approvals the procedure will need.
• Start teaching, Stop, and Cancel run on the bot's real computer through the human-control boundary owned by AKR-106. Observation is explicit and visible, and the bot does not act while the user demonstrates.
• Capture is a bounded, ordered record of action context, good enough to derive a procedure. Passwords, secrets, and protected-input values are removed before anything is stored or sent to a model. Retention and deletion of captures are explicit.
• The bot produces editable steps, the required sites, connectors, and skills, parameters, consequential actions, and uncertainties. It asks concrete questions instead of inventing a step it did not observe.
• Saving needs explicit approval and goes into the existing routine model, reusing dry runs, approval invalidation after edits, dependency health, schedules, and run history. A demonstration never turns on a schedule by itself.
• Corrections update the draft and require approval again. Cancel or discard removes the draft and releases control, leaving no scheduled routine.
• Send, pay, delete, production, and secret actions still need approval at execution time.
• Web and Electron provide the teaching controls and the review. Native mobile shows the resulting routine safely. Starting a teaching session through native mobile remote-desktop takeover is unselected feature 2 and is not part of this issue.

Launch examples, as user outcomes rather than separate dashboards: inbox triage, a sourced research brief, a cheaper-shopping comparison, and one approved booking.

Acceptance criteria

• Demonstrate a deterministic multi-step workflow, stop capture, inspect and edit the derived steps, approve, dry-run, then replay from fresh fixture state with the expected result
• Editing a saved procedure requires approval again. Pause, resume, and delete keep working in the existing routine editor and bot panel.
• Cancel mid-capture, a disconnect, a failed draft generation, unavailable connectors, and expired computer control all recover without replaying actions or storing secrets
• The inbox, research, shopping, and booking examples are run as acceptance scenarios on isolated fixtures, and any real connector that is unavailable is reported as such instead of claimed
• Focused tests cover observation ordering, redaction, approval boundaries, draft persistence, and routine integration, using receipts and drains, not sleeps
• The primary agent runs an isolated end-to-end web and Electron pass at desktop and narrow widths across teaching, the routine editor, the bot panel, and history
• Bot provider compatibility is checked explicitly, docs/user/ and docs/internals/ are updated, and targeted lint and typechecks pass

How to verify

Run the focused teaching and routine tests. Then, in an isolated dev environment with a bot on a workspace that supports human control, start teaching, demonstrate a fixture workflow, stop, edit and approve the draft, dry-run it, and replay it from fresh fixture state. Repeat with cancel mid-capture and with an expired control session. Check the screens at desktop and narrow widths on web and Electron, and the saved routine on mobile.

Out of scope

• A second scheduler or a visual workflow language
• Background recording of the user's own live desktop
• Automatic approval of send, pay, delete, production, or secret actions
• A second bot-browser stack, or presenting fixture capture as real teaching
• Pushing, publishing, or changing Linear state as part of the implementation

Context

Work can proceed in parallel. Leo asked agents to build the independent parts now instead of waiting on whole desktop and browser issues. AKR-106 (watch and take control of a bot's remote desktop) and AKR-92 (drive the sandbox browser through the Akeru runtime) are integration requirements for observing and replaying real computer actions, not gates on starting. Build observation records, redaction, the teaching-session lifecycle, cancel and discard, procedure generation, the editable review, approval invalidation, and routine integration now, tested with deterministic observed-action fixtures. Reuse a real observation path if one already exists.

Ownership: AKR-106's owner owns the computer transport and control contracts. This issue owns teaching records, the review, and conversion to a routine. Avoid concurrent edits to shared files. Coordinate, or keep the proposed interface outside the repository until the integration owner supplies it.

Completion needs the real boundary. Real demonstration capture and replay need the actual authorized computer boundary and a full browser check. If that boundary or a credential is missing, report the exact remaining integration check and finish everything else. All approval, privacy, and verification requirements still apply. Historical project dates, parked milestones, issue status, and old model assignments do not block this work. Use the agent's own assigned worktree and its current AGENTS.md.

What main has today: human control exists through ComputerRegistry (apps/server/src/provider/computerRegistry.ts), ComputerViewerDialog, and the viewer controller's takeControl. It currently requires a graphical Daytona workspace on Codex or Kimi. ComputerActionReceipt (packages/contracts/src/computer.ts) records only an ordered category per action (click, move, scroll, key, type), with no text, keys, selectors, or coordinates, so it is not enough to derive a procedure on its own. request_box_help hands a human-only step to the user through the inbox. AKR-127 is reducing the sandbox providers to Local, E2B, Vercel, and a Cloudflare bridge, which drops Daytona, so the teaching boundary should not depend on Daytona.

Where to start: packages/contracts/src/routines.ts, apps/server/src/routines/, apps/server/src/provider/botBrowser.ts, apps/server/src/provider/AkeruSessionResources.ts, apps/web/src/browser/browserRecording.ts, packages/client-runtime/src/state/routines.ts, the bot routine editor, and packages/contracts/src/computer.ts.

History: Leo selected this as feature 3, teaching routines by demonstration, on 2026-09-07, and asked that this issue be reused rather than a second teaching system. The selection is not evidence that the feature has shipped. Earlier model routing for this issue: Fable for the review UI, Sol for the procedure contract and execution boundary, Grok as critic.

Relations: blocks AKR-81 (Pass the clean-install launch gate). Related: AKR-92, AKR-106, AKR-76, AKR-79, LEO-304, LEO-305, LEO-306, LEO-307 (the shipped routine work), LEO-249, LEO-294, LEO-301. Blocked by LEO-243 (approval cards) and LEO-290 (connector health), both done.

Created with Claude Opus 5.5 in Claude Code.