Cooperative UI turns for concurrent winapp ui agents - #767
Draft
Nikola Metulev (nmetulev) wants to merge 3 commits into
Draft
Cooperative UI turns for concurrent winapp ui agents#767Nikola Metulev (nmetulev) wants to merge 3 commits into
winapp ui agents#767Nikola Metulev (nmetulev) wants to merge 3 commits into
Conversation
Concurrent winapp.exe processes share one Windows desktop, so a command could steal focus or dismiss another workflow's transient UI even though the existing foreground guards stopped wrong-window injection. Add an owner-aware coordination service so desktop-driving commands take turns. Coordinator (Services/InteractiveDesktop): - Owner identity: WINAPP_UI_OWNER_ID, else the immediate parent process, else a one-command anonymous owner. Only a domain-separated SHA-256 key is persisted; the raw value never reaches disk, logs, or telemetry. - state.lock, atomically published state.json with schema versioning and unknown-field preservation, active.lock, and DeleteOnClose participant leases scoped per user and Windows session. - A pure, clock-injected scheduler implementing the forward barrier, FIFO promotion, four-second idle grace, and handoff. - Liveness is proven only by a held lease plus PID/start match: there are no heartbeats, so a suspended process keeps its place in the queue. Command integration: - New two-phase UiCoordinatedAction: all local validation runs in Preflight, so a malformed command never opens a lease, takes a ticket, or joins the queue. - All 21 ui commands declare a mode. active.lock is held only across the desktop-sensitive section, never across output formatting, PNG encoding or file publication. - Every DesktopExclusive command resolves and validates the HWND, PID and element it acts on inside that section, so nothing acts on state captured before an unbounded queue wait. - IDesktopForegroundService is now the only path to SetForegroundWindow and window restore. - send-keys no longer focuses its --target before foreground validation. - record is TurnShared so same-owner input interleaves; screenshot starts observational and escalates the whole invocation, discarding buffered captures and recapturing from the beginning. Also: privacy-minimized bucketed coordination telemetry, npm AbortSignal threaded through both spawn helpers, and documentation across the UI guide, usage, telemetry, JSON envelope reference, shipped skill, agent guidance, sample, and npm README. Refs #764
…ce tests Review round 2 findings, each with a regression test proven to fail without the fix: 1. Prior-boot idle deadline stranded the turn. Environment.TickCount64 resets on reboot, so a persisted deadline from a long-uptime session could hold the desktop for days. Normalize now clamps a deadline beyond now + IdleGraceMs, and the UTC diagnostic is overflow-safe. 2. Screenshot escalation swallowed OperationCanceledException and UiCoordinationException in its catch-all, reporting internal_error while the coordinator saw a normal completion and renewed the grace. All ten UI handlers now filter their catch-all with UiCoordinatedAction.IsCoordinationFault, and the coordinator no longer treats a cancelled token as a normal completion. 3. CompleteCommand rewrote the idle deadline without checking that the completing owner is the current owner, letting a foreign completion revoke or extend a stranger's grace. It now takes the full UiOwnerIdentity and mutates the deadline only on a match. 4. RegisterObserve ignored a Detached admission, leaving the lease open and completing against a foreign owner when ownership lapsed mid-registration. 5. IDesktopSection.EnterAsync documented reentrancy the implementation deliberately does not provide. 6. Missing state.json was treated as fresh unconditionally, so an external deletion while a participant was live could mint a second owner. It is fresh only with no live participant and a free active.lock. Also adds spec 18.3 real-app acceptance coverage: a tight burst protecting transient menu UI, a >4s reasoning gap forcing handover and replay, and a recording pinning its owner while same-owner input continues and another owner waits - driven by real separate winapp.exe processes against a real window. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
…daries Recording/cancellation correction (reviewer): - bodyCompletedNormally now means "the body returned rather than threw", not "the token is unset". `ui record` observes Ctrl+C deliberately, finalizes the MP4 and returns success; per spec that is a completed command and must renew the owner's grace. The earlier generic token check wrongly denied renewal. Commands that must not renew let cancellation propagate instead, which is what the handler catch-all filters are for. - Added coverage on both sides: a queued screenshot escalation that is cancelled removes its ticket and renews nothing, while an active recording that finalizes on cancellation does renew. - Added handler-level coverage that UiScreenshotCommand lets a cancelled or refused escalation escape instead of flattening it to internal_error. Additional hardening: 1. InteractiveDesktopPaths.IsCurrentUserOnly now also requires the directory owner to be the current user: an owner implicitly holds WRITE_DAC and can rewrite even a protected, current-user-only DACL. The repair path re-reads and fails closed if ownership could not actually be taken. 2. UiSendKeysCommand now runs the same in-section HWND/PID validation as click and invoke. Without --target it could act on a recycled session handle; with --target the re-resolved element is verified to still belong to the session process before foreground, focus, post or send. 3. Lock acquisition no longer retries every IOException. Only ERROR_SHARING_VIOLATION (32) and ERROR_LOCK_VIOLATION (33) are contention; anything else fails desktop_coordination_unavailable instead of spinning forever on a failure that will never clear. ParticipantRegistry keeps its fail-safe-as-live probe (documented why it differs) but no longer reports a vanished lease as held. 4. Screenshot blank-retry foreground now goes through IDesktopForegroundService rather than a direct PInvoke.SetForegroundWindow seam, so every foreground change has one choke point. Added a guard test that scans production sources for direct SetForegroundWindow/ShowWindow calls outside that service, plus a self-check that the guard's pattern still matches. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| /// </remarks> | ||
| private readonly SemaphoreSlim _sectionGate = new(1, 1); | ||
|
|
||
| private IParticipantLease? _lease; |
Comment on lines
+536
to
+543
| foreach (var command in state.OwnerCommands) | ||
| { | ||
| if (command.Mode == UiTurnMode.DesktopExclusive && command.Ticket is { } ticket | ||
| && (earliestBarrier is null || ticket < earliestBarrier)) | ||
| { | ||
| earliestBarrier = ticket; | ||
| } | ||
| } |
Comment on lines
+138
to
+144
| foreach (var leaseFile in leaseFiles) | ||
| { | ||
| if (IsLeaseFileHeld(leaseFile)) | ||
| { | ||
| return true; | ||
| } | ||
| } |
| { | ||
| _sessionToken = processInspector.CurrentSessionId.ToString(CultureInfo.InvariantCulture); | ||
| LockDirectory = ResolveLockDirectory(); | ||
| ParticipantsDirectory = Path.Combine(LockDirectory, "participants"); |
| _sessionToken = processInspector.CurrentSessionId.ToString(CultureInfo.InvariantCulture); | ||
| LockDirectory = ResolveLockDirectory(); | ||
| ParticipantsDirectory = Path.Combine(LockDirectory, "participants"); | ||
| StateLockPath = Path.Combine(LockDirectory, $"{FilePrefix}{_sessionToken}.state.lock"); |
| LockDirectory = ResolveLockDirectory(); | ||
| ParticipantsDirectory = Path.Combine(LockDirectory, "participants"); | ||
| StateLockPath = Path.Combine(LockDirectory, $"{FilePrefix}{_sessionToken}.state.lock"); | ||
| StatePath = Path.Combine(LockDirectory, $"{FilePrefix}{_sessionToken}.state.json"); |
| ParticipantsDirectory = Path.Combine(LockDirectory, "participants"); | ||
| StateLockPath = Path.Combine(LockDirectory, $"{FilePrefix}{_sessionToken}.state.lock"); | ||
| StatePath = Path.Combine(LockDirectory, $"{FilePrefix}{_sessionToken}.state.json"); | ||
| ActiveLockPath = Path.Combine(LockDirectory, $"{FilePrefix}{_sessionToken}.active.lock"); |
Comment on lines
+96
to
+99
| => Path.Combine( | ||
| ParticipantsDirectory, | ||
| $"{FilePrefix}{_sessionToken}-{processId.ToString(CultureInfo.InvariantCulture)}-" + | ||
| $"{ProcessInspector.FormatStartTicks(startTicksUtc)}{LeaseExtension}"); |
| } | ||
|
|
||
| return ValidateLockDirectory( | ||
| Path.Combine(localAppData, "Microsoft", "WinAppCli", "locks"), |
Comment on lines
+289
to
+291
| var quarantinePath = System.IO.Path.Combine( | ||
| paths.LockDirectory, | ||
| $"state.corrupt-{clock.UtcNow.ToString("yyyyMMdd'T'HHmmss'.'fff'Z'", CultureInfo.InvariantCulture)}.json"); |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds owner-aware cooperative turns so concurrent winapp ui workflows safely share the Windows desktop.
Changes:
- Adds lease-backed scheduling, desktop locking, recovery, cancellation, and telemetry.
- Coordinates all UI commands with post-wait target revalidation.
- Adds npm
AbortSignalsupport, documentation, and extensive tests.
Reviewed changes
Copilot reviewed 87 out of 87 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
src/winapp-npm/test/abort-signal.test.ts |
Tests abort propagation. |
src/winapp-npm/src/winapp-commands.ts |
Adds generated signal support. |
src/winapp-npm/src/winapp-cli-utils.ts |
Forwards abort signals. |
src/winapp-npm/src/ui-record-guard.ts |
Forwards recording signals. |
src/winapp-npm/scripts/generate-commands.mjs |
Generates signal options. |
src/winapp-npm/README.md |
Documents cancellation and ownership. |
docs/npm-usage.md |
Adds generated signal documentation. |
src/winapp-CLI/WinApp.Cli/Services/InteractiveDesktop/* |
Implements coordination state, scheduling, leases, locking, recovery, output, and telemetry. |
src/winapp-CLI/WinApp.Cli/Services/UiAutomationService.cs |
Injects foreground coordination. |
src/winapp-CLI/WinApp.Cli/Services/UiAutomationService.Screenshot.cs |
Coordinates screenshot escalation. |
src/winapp-CLI/WinApp.Cli/Services/UiAutomationService.Record.cs |
Coordinates recording operations. |
src/winapp-CLI/WinApp.Cli/Services/IUiAutomationService.cs |
Extends capture contracts. |
src/winapp-CLI/WinApp.Cli/Helpers/UiCoordinatedAction.cs |
Adds coordinated command lifecycle. |
src/winapp-CLI/WinApp.Cli/Helpers/IDesktopForegroundService.cs |
Centralizes foreground operations. |
src/winapp-CLI/WinApp.Cli/Helpers/DesktopTargetValidation.cs |
Validates targets after waiting. |
src/winapp-CLI/WinApp.Cli/Helpers/PointerCommandSupport.cs |
Routes foreground requests safely. |
src/winapp-CLI/WinApp.Cli/Helpers/UiJsonError.cs |
Adds coordination errors. |
src/winapp-CLI/WinApp.Cli/Helpers/UiJsonContext.cs |
Serializes coordination details. |
src/winapp-CLI/WinApp.Cli/Helpers/HostBuilderExtensions.cs |
Registers coordination services. |
src/winapp-CLI/WinApp.Cli/Commands/Ui*Command.cs |
Classifies and coordinates UI commands. |
src/winapp-CLI/WinApp.Cli/Program.cs |
Opens telemetry scope. |
src/winapp-CLI/WinApp.Cli/NativeMethods.txt |
Adds process-enumeration APIs. |
src/winapp-CLI/WinApp.Cli/Telemetry/Events/CommandCompletedEvent.cs |
Reports coordination telemetry. |
src/winapp-CLI/WinApp.Cli.Tests/InteractiveDesktopSchedulerTests.cs |
Tests scheduler semantics. |
src/winapp-CLI/WinApp.Cli.Tests/InteractiveDesktopStoreTests.cs |
Tests storage and recovery. |
src/winapp-CLI/WinApp.Cli.Tests/InteractiveDesktopLockTests.cs |
Tests lock lifecycle. |
src/winapp-CLI/WinApp.Cli.Tests/InteractiveDesktopMultiprocessTests.cs |
Tests cross-process coordination. |
src/winapp-CLI/WinApp.Cli.Tests/InteractiveDesktopRealAppTests.cs |
Tests real desktop workflows. |
src/winapp-CLI/WinApp.Cli.Tests/UiCommandTests.Coordination.cs |
Tests command classifications. |
src/winapp-CLI/WinApp.Cli.Tests/FakeInteractiveDesktopLock.cs |
Provides fake turns. |
src/winapp-CLI/WinApp.Cli.Tests/FakeDesktopForegroundService.cs |
Provides fake foreground operations. |
src/winapp-CLI/WinApp.Cli.Tests/DesktopPrimitiveGuardTests.cs |
Guards foreground-call conventions. |
src/winapp-CLI/WinApp.Cli.Tests/FakeUiServices.cs |
Extends UI test fakes. |
src/winapp-CLI/WinApp.Cli.Tests/UiCommandTests.cs |
Registers coordination fakes. |
src/winapp-CLI/WinApp.Cli.Tests/UiCommandTests.Record.Stdin.cs |
Updates handler construction. |
src/winapp-CLI/WinApp.Cli.Tests/UiAutomationServicePureTests.cs |
Tests coordinated blank retries. |
src/winapp-CLI/WinApp.Cli.Tests/RealUiAutomationTests*.cs |
Updates capture and recording tests. |
src/winapp-CLI/WinApp.Cli.Tests/GestureTargetingTests.cs |
Updates service test contract. |
src/winapp-CLI/WinApp.Cli.Tests/UiaTestFixture.cs |
Adds transient-menu fixtures. |
docs/ui-automation.md |
Documents cooperative turns. |
docs/usage.md |
Documents workflow identity. |
docs/telemetry.md |
Documents coordination telemetry. |
plugins/winapp/skills/winapp-ui-automation/SKILL.md |
Updates agent UI guidance. |
plugins/winapp/skills/winapp-ui-automation/references/ui-json-envelope.md |
Documents coordination errors. |
plugins/winapp/agents/winapp.agent.md |
Adds ownership guidance. |
samples/winui-app/README.md |
Demonstrates owner setup. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| }; | ||
|
|
||
| var result = await uiAutomation.RecordAsync(session, selector, options, linkedCts.Token, OnRecordingStarted); | ||
| var result = await uiAutomation.RecordAsync(session, selector, options, turn, linkedCts.Token, OnRecordingStarted); |
| | `invalid_ui_owner_id` | `WINAPP_UI_OWNER_ID` is set but empty/whitespace or longer than 256 characters. Fails before any UI side effect. | | ||
| | `desktop_coordination_unavailable` | Coordination state could not be read, published, or safely rebuilt — including state written by a newer `winapp`. Mutating commands fail closed rather than acting uncoordinated. | | ||
| | `queue_capacity_exceeded` | 64 commands are already waiting for the desktop. | | ||
| | `cancelled` | Ctrl+C (or an npm `AbortSignal`) while the command was still waiting for its turn. The command never ran, so it has no UI side effects. Exit code **130**. | |
| effectiveOutcome, | ||
| WaitedMs, | ||
| _observedQueueDepth, | ||
| _waitWatch.ElapsedMilliseconds)); |
Comment on lines
+58
to
+59
| /// <summary>The command acquired the turn after another owner's idle grace expired.</summary> | ||
| HandoffAfterIdle, |
| | `profile` | `string \| undefined` | No | Certificate profile name. Must be used with --account | | ||
| | `resourceGroup` | `string \| undefined` | No | Resource group to narrow down signing accounts | | ||
| | `subscription` | `string \| undefined` | No | Azure subscription ID to use. If not provided and multiple subscriptions exist, you will be prompted. | | ||
| | `signal` | `AbortSignal \| undefined` | No | Cancels the whole native invocation, not just a wait for the shared desktop. |
Contributor
Build Metrics ReportBinary Sizes
Test Results✅ 4694 passed, 16 skipped out of 4710 tests in 631.5s (+129 tests, +17.6s vs. baseline) Test Coverage✅ 88.8% line coverage, 82.2% branch coverage · CLI Startup Time50ms median (x64, Try This BuildInstalls the MSIX for your architecture, replacing any previously installed build. Needs the GitHub CLI — the command offers to install it and sign you in if it is missing. & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) 767Switching between builds often?Put the tool on your PATH once: & ([scriptblock]::Create((irm https://raw.githubusercontent.com/microsoft/winappCli/main/scripts/winapp-pr.ps1))) -AddToPathThen this build is just: winapp-pr 767Run Updated 2026-08-18 17:16:02 UTC · commit |
Nikola Metulev (nmetulev)
marked this pull request as draft
August 18, 2026 18:25
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #764.
Multiple
winapp.exeprocesses can drive one signed-in Windows desktop, but that desktop has a single foreground window, a single focus, a single cursor and a singleSendInputqueue. Today two agents working at once silently corrupt each other's work: one steals the foreground mid-gesture, another's transient menu is dismissed before it can be clicked, keystrokes land in the wrong window. This adds owner-aware cooperative turns so concurrent agents interleave safely, with no opt-out — coordination is always on.Design
Each command declares one of three modes:
Observeui inspect,ui find,ui get-propertyTurnSharedui recordDesktopExclusiveui click,ui invoke,ui send-keys,ui drag, …An owner is a workflow, not a process: an explicit
WINAPP_UI_OWNER_ID, else the parent shell, else anonymous. Consecutive commands from one agent join the same turn, so a burst of related steps is atomic against other agents. A turn is released 4 seconds after the owner's last command finishes — a "reasoning gap" longer than that hands the desktop to whoever is waiting, which is intended: an idle agent must not pin the desktop.Coordination state lives in
%LOCALAPPDATA%behindstate.lock, withactive.lockguarding the desktop-sensitive section and oneDeleteOnCloselease per participant.Decisions worth reviewing
FileShare.Nonelease is the proof of life. A suspended process keeps its lease and therefore keeps its place at the head of the queue — deliberately, since a debugger-paused agent has not abandoned its work.UiCoordinatedActionsplits every handler intoPreflight(local-only) andExecuteAsync, so a malformed command never takes a ticket or joins a queue.active.lockdoes not wrap the whole handler — only the moment the command touches the shared desktop. Output formatting, PNG encoding and file publication stay outside it.state.jsonstores only a domain-separated SHA-256, and--verbosereports PIDs and queue depth, never target app or window text.130with a structuredcancelledpayload. A silent timeout would produce exactly the half-finished UI this feature exists to prevent.Behavior and compatibility
Existing JSON contracts are unchanged; four new error codes (
desktop_coordination_unavailable,cancelled,invalid_owner_id,queue_capacity_exceeded) and an optionalcoordinationblock are additive.ui screenshotstill runs observationally and escalates to an exclusive turn only if a target actually needs restore or foreground, discarding buffered captures so a published image never mixes pre- and post-escalation pixels. The npm wrapper gainsAbortSignalsupport.Test evidence
winapp.exeprocesses queueing against the file protocol. This cannot be simulated in one process: participant identity is(pid, processStartTicks), so two "owners" inside one process share an identity and a lease.winapp.exeprocesses: a tight burst protects transient menu UI while another owner waits; a >4s reasoning gap hands over the turn and forces replay; a recording pins its owner while same-owner input continues and a different owner's mutation waits.SetForegroundWindow/ShowWindowoutsideIDesktopForegroundService, so the "one choke point" invariant holds by construction rather than convention.Every correctness fix from review was validated by disabling the fix and confirming the corresponding test fails.
Known environmental failures (pre-existing, unrelated)
Mp4SinkWriterEncoder_RealEncoderCoversValidationAndSuccessfulComplete— fails on arm64 with and without these changes.Record_WebView2StyleCompositedContent_InteractiveCapturesFrames— the test invokes a framework-dependentwinapp.exeand fails withhostpolicy.dll not foundbefore any winapp code runs. A test-harness issue; can be filed separately.api.nuget.org, which is unreachable from corp machines (documented inAGENTS.md).Docs
docs/ui-automation.md,usage.md,telemetry.md,npm-usage.md,ui-json-envelope.md, the shipped UI skill and agent guidance, the WinUI sample README, and the npm README are updated.docs/cli-schema.jsonregenerates byte-identical, confirming no CLI surface change.No PR-review findings remain outstanding.