[Feature]: Cooperative UI turns for concurrent winapp UI agents

[nmetulev]

Is your feature request related to a problem? Please describe.

Multiple agents can run separate winapp ui processes against different applications in the same signed-in Windows session. Windows exposes only one foreground window, keyboard focus, cursor, and SendInput stream, so commands can interfere even when they target different apps.

Examples:

Agent A opens a transient menu in Notepad
Agent B foregrounds Calculator
-> Notepad loses focus and closes the menu
-> Agent A's next command fails because the menu item disappeared

Agent A moves the cursor to click a button
Agent B foregrounds another app and sends keys
-> commands fail foreground validation or act on stale UI state

Existing foreground guards correctly prevent many wrong-window injections, but they do not coordinate processes. A command may change focus or dismiss another workflow's transient UI before it later fails safely.

Local empirical tests with separate winapp.exe processes showed:

Scenario	Result
Transient menu workflow running alone	20/20 succeeded
Same workflow with no coordination	0/20 succeeded
Same workflow with per-command exclusion only	0/20 succeeded
Same workflow with workflow ownership	19/20 succeeded
35-second real Notepad workflow, no hard cap	3/3 succeeded
Same workflow with a forced 30-second handoff	0/3 succeeded

Separate agent tool calls used fresh shell PIDs, while commands inside one script reused one shell PID. This supports short script-level turns that naturally expire during model reasoning.

Describe the solution you'd like

Add one owner-aware IInteractiveDesktopLock coordination service with three command modes:

Mode	Behavior
Observe	A non-owner runs concurrently. If the current owner runs it, the command pins and renews that owner's existing turn.
TurnShared	Claims or waits for the workflow turn. Same-owner shared work may overlap. Used by recording.
DesktopExclusive	Claims or waits for the turn, creates a forward barrier for later same-owner participating commands, and exclusively uses foreground/focus/cursor/input while acting.

Example:

Owner A record  --------------------------------------
Owner A click             [DesktopExclusive]
Owner A later invoke             waits, then runs
Owner B click      waits for Owner A's workflow turn
Owner B inspect   runs concurrently

Owner identity

Resolve one logical workflow owner in this order:

1. Explicit WINAPP_UI_OWNER_ID.
2. Immediate parent PID plus parent process start time, for tight commands issued by one long-lived shell/script.
3. Anonymous one-command owner if parent inspection fails.

WINAPP_UI_OWNER_ID identifies a logical UI workflow, not necessarily an entire agent or one app. Cooperating parallel processes use the same value; independent workflows use different values. Persist only a SHA-256 owner key, never the raw value.

Turn lifetime

• Fixed four-second idle grace after the latest non-cancelled owner command completes.
• No hard turn cap while owner commands continue to run or arrive within the grace.
• The four-second grace protects tight scripted bursts; it intentionally expires during model reasoning.
• Adaptive workflows must reacquire, revalidate, and replay transient setup after a reasoning gap.
• Other participating owners wait FIFO until acquisition or cancellation.

Local coordination files

Scope files by current user profile and Process.SessionId:

%LOCALAPPDATA%\Microsoft\WinAppCli\locks\
  interactive-desktop-{session}.state.lock
  interactive-desktop-{session}.state.json
  interactive-desktop-{session}.active.lock
  participants\interactive-desktop-{session}-{pid}-{startTicks}.lease

• state.lock: short FileShare.None lock around every state read/update.
• state.json: owner, active commands, four-second expiry, owner-local tickets, and global FIFO waiters. Publish through flushed same-directory temporary files and atomic replacement.
• active.lock: long FileShare.None lock around desktop-sensitive sections.
• Participant leases: process-held FileShare.None + FileOptions.DeleteOnClose liveness proof. No periodic heartbeat writes.

Participant ordering is strict: open the lease before publishing participation; remove the state entry before closing the lease.

Command behavior

Proposed initial classification:

Observe:
  status, list-windows, inspect, search, get-*, wait-for,
  set-value, UIA scroll, ordinary background screenshot

TurnShared:
  record

DesktopExclusive:
  invoke, click, drag, hover, wheel scroll, touch, pen,
  focus, send-keys, focused/screen screenshot paths

DesktopExclusive is a forward barrier:

• Earlier running TurnShared commands may continue.
• Later TurnShared and DesktopExclusive commands wait.
• Observe commands remain concurrent.

Every desktop-exclusive command must reacquire and revalidate HWND, PID, selector/element, bounds/coordinates, and foreground state after queue waiting. No action may use stale pre-wait target state.

send-keys --target should be reordered to avoid disturbing another workflow before foreground validation:

resolve target without focusing
-> acquire turn and active lock
-> revalidate
-> request and verify top-level foreground
-> focus child control
-> send input

Every recording claims TurnShared ownership before capture begins. It pins that owner's turn, while same-owner clicks/input may continue. It takes active.lock only for restore/foreground or equivalent desktop-sensitive moments.

A normal screenshot begins as Observe. If any target requires restore/foreground, escalate the whole invocation to DesktopExclusive, discard buffered captures, queue, rediscover/revalidate every target, and recapture from the beginning before publishing output.

Waiting and cancellation

• Waiting is indefinite and cancellable; there is no coordination timeout in v1.
• Default TTY output shows a simple waiting status after one second.
• --verbose also shows local parent/active PIDs, operation, queue depth, commands ahead, and elapsed wait.
• --quiet is silent.
• --json remains silent until the final result or structured cancellation/error.
• Native cancellation while queued removes the ticket and exits 130 with error code cancelled.
• Cancellation after acquisition preserves existing command semantics; for example, active recording Ctrl+C still finalizes normally.
• npm adds signal?: AbortSignal; on Windows this is whole-call force cancellation, and stale queue state is reclaimed through participant-lease/PID pruning.

Recovery and compatibility

• Never steal active.lock from a live process.
• Windows process termination releases file handles and deletes participant leases.
• Atomic state publication leaves either the old complete state or new complete state after a crash.
• Recover corrupt state only when active.lock is free and no live participant lease exists; otherwise fail closed.
• Unknown newer state versions are not reset. Participating/mutating commands fail desktop_coordination_unavailable; detached observations may continue.
• Coordination is guaranteed only among compatible updated binaries.

Additional context

Accepted tradeoffs:

• A healthy long script, unbounded recording/wait, failure loop, or hung live process can block other participating owners indefinitely.
• A live suspended waiter at the FIFO head can block later waiters until resumed or terminated.
• Non-owner Observe mutations may change the same app while another owner works; this feature prevents desktop interference, not transactional app-state isolation.
• Physical users and third-party tools remain outside coordination.
• App behavior occurring asynchronously after invoke returns is outside active.lock, although turn ownership/grace still applies.
• npm force cancellation may leave partial recording output.

Key acceptance criteria:

• Zero overlapping DesktopExclusive sections among compatible updated processes.
• Zero wrong-window OS-wide input in coordination tests.
• FIFO ordering comes from persisted tickets, not file-open race order.
• Tight open/inspect/act bursts preserve transient UI.
• Adaptive reasoning gaps hand off after four seconds and require replay.
• Recording pins the owner while same-owner input continues.
• Queued desktop commands revalidate changed/closed targets before acting.
• Killing processes at every registration/removal transition leaves no stranded ownership.
• Corrupt state is never reset over a live participant.
• No raw owner identity or PID enters telemetry.

Implementation would include the complete coordinator, command integration, foreground helper centralization, cancellation/output/telemetry, npm AbortSignal, documentation, deterministic state-machine tests, multiprocess tests, and real-app acceptance tests in one PR.

[nmetulev]

Full spec Cooperative UI Turns for winapp ui.txt