feat(windows): harden and productionize the maka-cu Computer Use executor

Relationship

Part of #2142, Phase 5 (Windows Computer Use).

Context

The Windows support roadmap currently lists Computer Use as deferred work: define a Windows backend using UI Automation plus an appropriate capture API, design consent/secure-desktop/elevation/multi-monitor/scaling/session-lock behavior, reuse the platform-neutral Computer Use host event contract, and add Windows integration/E2E evidence.

This issue proposes the concrete executor work needed to make that phase implementable and reviewable. It is based on a code review of maka-agent/maka-cu, revision 4a9787d, especially its experimental apps/OpenComputerUseWindows runtime.

The current implementation is useful as a functional prototype: Go owns the CLI/MCP/tool schema and an in-process snapshot cache; an embedded PowerShell bridge uses Windows UI Automation for discovery/tree rendering/semantic actions and falls back to Win32 window messages for some input paths. It has been validated against basic Notepad flows. It is not yet a safe or reliable equivalent of the macOS maka.cu/2 executor.

Review Findings

1. Observation/action binding is too weak

The Windows runtime caches snapshots by a lower-cased app query/name/bundle-like process name/PID and action calls reuse a numeric element_index. The PowerShell bridge then re-enumerates the current process tree. If a UIA runtime id is unavailable, it falls back to the first matching AutomationId/name and control type:

This can target the wrong window/control after an app restart, process recycling, modal-window creation, duplicate controls, or a reflow. The macOS host protocol already has the stronger model: opaque per-snapshot element tokens, snapshot state (live, spent, superseded, expired, evicted), process identity, window identity, and an element digest:

2. The screenshot path can disagree with the UIA tree

Capture-WindowPngBase64 uses Graphics.CopyFromScreen on the window rectangle:

runtime.ps1#L565-L632

When the target is covered by another window, the image contains the occluding window while the tree describes the target app. Minimized, layered, hardware-accelerated, and some redirected windows can also produce incomplete or black captures.

The production path should prefer a target-window capture API such as Windows Graphics Capture (IGraphicsCaptureItemInterop::CreateForWindow(HWND, ...)) and report an explicit degraded capability when only a screen-rectangle fallback is available. PrintWindow can be evaluated as a compatibility fallback, but it is synchronous and application-dependent, so it must not be treated as universally correct.

3. Win32 input currently reports success without verification

The bridge ignores PostMessage return values and returns ok=true after a fixed 120 ms delay:

A message can fail because of UIPI/integrity level, an invalid or recycled HWND, a toolkit that ignores the message, or a target that has not processed it yet. Windows documents that PostMessage and SendInput are subject to UIPI. The executor must distinguish at least:

refused before dispatch;
dispatched and verified;
dispatched but outcome unknown;
dispatch failed;
unsupported for this toolkit/window.

It must never silently turn an explicitly background-safe path into foreground SendInput or global pointer input.

4. The wheel fallback uses the wrong coordinate space

Send-Scroll converts the target point to client coordinates before putting it in WM_MOUSEWHEEL.lParam:

runtime.ps1#L188-L203

WM_MOUSEWHEEL expects screen coordinates in lParam, unlike ordinary client-area mouse messages. This is incorrect for a window not positioned at the screen origin and is especially visible on multi-monitor layouts with negative coordinates.

5. DPI, threading, and per-call process startup need a production decision

The UIA bounding rectangle is in physical screen coordinates, while the runtime does not establish a clear Per-Monitor-V2 DPI contract for the executor and coordinate conversion. Mixed-DPI monitors can therefore make the screenshot, UIA frame, and Win32 input disagree.

Every tool call currently writes a temporary script and starts a new Windows PowerShell process. This is simple for a prototype but expensive and makes long-lived UIA element/cache/event ownership difficult. Microsoft recommends using a dedicated non-UI MTA thread for UI Automation clients and provides cache requests to reduce cross-process property calls.

Proposed Direction

A. Reuse the platform-neutral host contract

Windows should implement the same native host protocol used by macOS (maka.cu/2) rather than growing a second, Windows-only nine-tool executor contract. The Maka runtime/host remains responsible for model-facing Anthropic Computer Use semantics; the native executor remains responsible for observation, target binding, dispatch, capture, and verification.

The protocol should carry:

executor version and capabilities/limits;
session lifecycle and cancellation;
observe results with opaque snapshot IDs and element tokens;
target identity including PID, process start time, and HWND/window generation;
element/window digests and explicit stale/unknown/expired/spent errors;
dispatch result fields for outcome, tier, path, effect, and verification;
an explicit capability/degraded result for missing capture, locked desktop, secure desktop, elevation/UIPI, or toolkit limitations.

B. Use stable target identity, not app-name identity

The minimum Windows target identity should be:

session + snapshotId + pid + processStartTime + hwnd + windowGeneration

HWND must be revalidated at dispatch time (IsWindow, owning PID, and current process start time). A recycled PID or HWND must fail closed. Element tokens must be opaque and resolved only inside the quoted snapshot; numeric indexes can remain a display convenience for the model/runtime, never the dispatch authority.

C. Make capture and input capability-driven

Recommended dispatch tiers:

UIA semantic pattern (Invoke, Toggle, SelectionItem, Value, Scroll, Text);
target HWND/window-message path, only when the target control and message contract are known;
Windows Graphics Capture / target-window coordinate path;
foreground SendInput, only as explicit opt-in with user-visible policy and verification.

Each result should identify the selected path and whether the operation was verified. Unsupported or unsafe paths should return typed errors, not fallback silently.

D. Prefer a long-lived native Windows bridge

Keep the Go/Node integration boundary if useful, but replace per-call PowerShell startup with a long-lived C#/.NET or native helper. It should own:

UIA COM initialization on a dedicated MTA worker;
cache requests for bulk tree properties;
event-driven invalidation for window/control changes;
Windows Graphics Capture sessions;
Win32/DPI/monitor identity and coordinate conversion;
structured HRESULT/Win32/UIPI errors.

A C#/.NET helper is likely the lowest-risk first production step; a direct Go COM/WinRT implementation can be evaluated later if packaging and maintenance justify it.

E. Define the Windows security/session contract

The implementation must explicitly handle and test:

normal interactive desktop versus service/SSH/session-0 execution;
locked workstation and unavailable input desktop;
UAC secure desktop and elevated target processes;
UIPI/integrity-level mismatches;
multi-monitor and mixed-DPI layouts;
minimized, occluded, redirected, and hardware-accelerated windows;
sensitive apps and credential/password fields;
screenshot and input consent/approval;
no automatic app launch, focus stealing, or foreground fallback by default.

The product must surface unsupported/deferred Computer Use capability in the Windows preview instead of claiming parity.

Proposed Deliverables

Acceptance Criteria

Protocol and safety

An action planned against snapshot A cannot execute against snapshot B, a recycled PID, or a recycled HWND.
A spent, expired, superseded, evicted, unknown, or mismatched snapshot returns a distinct typed result.
No unsupported path silently falls back to foreground/global input.
Every action result states the attempted path, outcome, effect, and verification status.
Locked/secure desktop/UIPI/elevated-target conditions are reported explicitly.

Observation and capture

UIA tree and screenshot refer to the same PID/HWND/window generation.
Target-window capture is correct when another window covers the target, or the result explicitly declares the degraded capture mode.
Coordinates remain correct under Per-Monitor-V2 scaling, mixed-DPI monitors, and negative virtual-screen coordinates.
Tree rendering has bounded node/depth/time budgets and does not hang on a provider that stops responding.

Actions

Semantic actions work without foreground activation for supported Win32, WPF/WinUI, Electron/WebView2, and browser fixtures where the toolkit exposes the required pattern.
Message/input fallbacks have return-value and post-action verification; failures are not reported as success.
type_text, key combinations, click, drag, horizontal/vertical scroll, and set-value have explicit capability coverage per toolkit.

Release evidence

A Windows 11 x64 CI lane runs protocol, fixture, and packaged smoke tests.
Interactive desktop tests cover Notepad, a Chromium/Electron/WebView2 fixture, modal windows, app restart, occlusion, locked session, elevation/UIPI, two monitors, and mixed DPI.
The Windows preview documentation states exactly what is supported, what is foreground-only, and what remains deferred.

Non-goals

This issue does not make every Windows GUI toolkit background-controllable.
This issue does not require global physical mouse movement as the default path.
This issue does not make the current PowerShell prototype production-ready by adding more heuristics alone.
This issue does not claim that Windows Computer Use is supported before the acceptance evidence exists.

References

Parent roadmap: #2142
Executor: maka-agent/maka-cu
Windows prototype: apps/OpenComputerUseWindows
macOS host protocol: OpenComputerUseKit/HostProtocol
Microsoft UI Automation threading guidance: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-threading
Microsoft UI Automation caching guidance: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-cachingforclients
Microsoft UI Automation scaling guidance: https://learn.microsoft.com/en-us/windows/win32/winauto/uiauto-screenscaling
Microsoft Windows Graphics Capture window interop: https://learn.microsoft.com/en-us/windows/win32/api/windows.graphics.capture.interop/nf-windows-graphics-capture-interop-igraphicscaptureiteminterop-createforwindow
Microsoft WM_MOUSEWHEEL coordinate contract: https://learn.microsoft.com/en-us/windows/win32/inputdev/wm-mousewheel
Microsoft PostMessage/UIPI behavior: https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-postmessagew

The five prototype gaps described here are real, and the Windows Computer Use problem is worth tracking. However, I would keep this issue as an epic rather than use the current checklist as one implementation contract. It currently combines a protocol redesign, durable helper lifecycle, UI Automation, capture, security/session policy, fallback input, toolkit coverage, DPI/multi-monitor behavior, and packaged E2E into one acceptance boundary. That boundary is too broad to close safely.

Three design changes are needed before implementation proceeds:

Do not make Windows pretend to implement the existing literal maka.cu/2 wire. That wire is a closed, macOS-specific contract: its paths, capture/session fields, permissions, and key semantics include AX/CG/Skylight, ScreenCaptureKit, TCC, and macOS modifiers. Windows should reuse the platform-neutral host boundary and the important invariants—bounded snapshots, opaque tokens, cancellation, fail-closed dispatch, and effect verification—while its child protocol remains private until there is a demonstrated need for a new public cross-platform revision. The current host only needs to stop assuming TCC-shaped preflight, a screenRecording lease, and Darwin-only selector names.
Separate observation from actuation. Windows Graphics Capture is a capture authority, not a dispatch tier. The result must independently report the capture path and the action path. In v1, capture should use CreateForWindow(HWND) and return capture_unavailable if that cannot be established; it must not silently fall back to a screen rectangle that may show a different window.
Define one narrow, testable vertical slice. I recommend the first child issue support only:
- an unlocked Windows 11 interactive desktop;
- one explicitly selected, already-running top-level HWND;
- a bounded UIA tree and WGC frame bound to the same validated process incarnation and HWND;
- set_value through ValuePattern;
- click_element only when the element explicitly advertises Invoke, Toggle, or SelectionItem;
- typed verified, refused, or unknown outcomes after re-observing the same HWND.

Every mutating action should spend its snapshot. Before and after the action, the helper should revalidate the owning PID's creation time, HWND, and retained UIA root. A modal or replacement HWND requires a new explicit selection. The tree and frame need to identify the same target generation; they do not need to claim impossible temporal atomicity for dynamic content.

The implementation should be a host-supervised long-lived child with one MTA UIA lane, bounded registry/tree/time budgets, cancellation, parent-death handling, and restart after a hung provider. It should reuse the existing product approval flow rather than introduce a second Windows consent state machine.

For v1, I would explicitly defer keyboard input, scrolling, window manipulation, PostMessage, SendInput, all coordinate paths, mixed-DPI/multi-monitor guarantees, and broad toolkit parity. Unsupported paths must refuse rather than fall through to global input. Evidence should be one deterministic adversarial fixture plus one packaged Windows 11 E2E that exercises the full observe → opaque token → action → readback chain.

With that boundary, this issue remains useful as the problem/epic record. The implementation should be split into smaller child issues, beginning with the bounded semantic preview above; the broader checklist should not be treated as the acceptance criteria for a single change.

简体中文

这里列出的五个原型缺口都真实存在，Windows Computer Use 也值得继续推进。不过，我建议把本 issue 保留为 epic，而不是把当前清单直接当成一个实现单的验收合同。当前边界同时包含协议重构、长驻 helper 生命周期、UI Automation、截图、安全/会话策略、输入 fallback、toolkit 覆盖、DPI/多屏和打包 E2E，范围过大，无法作为一个可安全闭合的交付。

实现前需要先调整三点设计：

Windows 不应冒充实现现有字面意义上的 maka.cu/2 wire。 该协议是闭集且明显面向 macOS，其中的路径、截图/会话字段、权限和按键语义包含 AX/CG/Skylight、ScreenCaptureKit、TCC 和 macOS 修饰键。Windows 应复用平台中立的 Host 边界，以及 bounded snapshot、opaque token、取消、fail-closed dispatch、效果验证等关键不变量；Windows child protocol 暂时保持私有，等确实需要新的公共跨平台 revision 时再引入。当前 Host 只需去掉对 TCC-shaped preflight、screenRecording lease 和 Darwin-only selector 的硬编码假设。
观察与执行必须分开。 Windows Graphics Capture 是 capture authority，不是 dispatch tier。结果应分别报告 capture path 和 action path。v1 截图只使用 CreateForWindow(HWND)；无法建立时返回 capture_unavailable，不能静默退化成可能截到其他窗口的屏幕矩形。
先定义一条窄而可验收的垂直切片。 首个子单建议只支持：
- 已解锁的 Windows 11 交互桌面；
- 一个由用户显式选择、已经运行的顶层 HWND；
- 绑定同一进程 incarnation 与 HWND 的 bounded UIA tree 和 WGC frame；
- 通过 ValuePattern 执行 set_value；
- 仅当元素明确声明 Invoke、Toggle 或 SelectionItem 时执行 click_element；
- 动作后重新观察同一 HWND，并只返回 typed verified / refused / unknown。

每个可能改变状态的动作都应消费 snapshot。动作前后都要重新校验进程创建时间、HWND 和保留的 UIA root；出现 modal 或新的 HWND 时必须重新显式选择。tree 与 frame 只承诺属于同一个 target generation，不宣称动态内容在时间上原子一致。

实现应为 Host 监管的长驻 child：单 MTA UIA lane、bounded registry/tree/time、取消、parent-death，以及 provider 卡死后的 kill/restart。继续复用现有产品 approval，不新增第二套 Windows consent 状态机。

v1 应明确后移 keyboard、scroll、window 操作、PostMessage、SendInput、所有坐标路径、mixed-DPI/多屏保证和广泛 toolkit parity。unsupported path 必须拒绝，不能 fallback 到全局输入。验收证据至少需要一个确定性 adversarial fixture，以及一条真实打包 Windows 11 E2E，覆盖 observe → opaque token → action → readback 全链路。

按这个边界，本 issue 可以继续作为问题/epic 记录；实现应拆成更小的子单，先交付上述 bounded semantic preview，而不是把当前大清单作为单个变更的验收标准。

Following the proposal above to start with a bounded Windows semantic preview, do maintainers have a preference for the native helper's implementation language: retain Go and replace the per-call PowerShell bridge, or implement a C#/.NET helper supervised directly by the existing TypeScript host?

The current apps/OpenComputerUseWindows prototype is Go + PowerShell, while the issue body suggests a long-lived C#/.NET or native helper. It would help to clarify the intended direction before implementing the first child issue.

The trade-offs as I currently understand them are:

Option	Advantages	Costs / risks
Keep Go; move native operations into a long-lived Go helper	Reuses the existing Go entry point and relevant CLI/MCP code; avoids adding a .NET build toolchain; can consolidate protocol handling and native execution in one helper.	UIA COM interfaces, WGC/WinRT capture, callbacks, and object lifetimes need suitable bindings and careful ownership. COM initialization and thread affinity must be designed explicitly rather than assuming goroutines provide the required execution context. Existing snapshot and dispatch behavior still needs redesign.
C#/.NET helper, called directly from TypeScript	.NET provides an established Windows SDK/interop path; explicit worker threads, typed results, and a persistent object registry are a practical fit for UIA and capture sessions. The host-to-helper path can remain simple without an intermediate Go forwarding process.	Requires porting the Windows execution logic and maintaining a .NET build/release lane. Framework-dependent publishing requires an installed runtime; self-contained publishing bundles it at a package-size and runtime-servicing cost. COM/WinRT interop, capture resource cleanup, provider hangs, and cancellation still need explicit engineering.

Keeping Go does not have to mean keeping per-call PowerShell. Making the PowerShell bridge persistent is another possible transition, but it would still need the same thread, event, registry, and lifecycle work. Conversely, a Go-to-C# bridge seems worth retaining only if the Go layer has CLI/MCP compatibility or other concrete responsibilities beyond forwarding messages.

Whichever language is chosen, I would keep the proposed v1 boundary unchanged: one explicitly selected HWND, UIA + target-window WGC observation, limited semantic actions, snapshot consumption, target revalidation, typed outcomes, and a supervised child that can be restarted after a hung provider. The choice should not require implementing the literal macOS maka.cu/2 wire or introducing foreground/global-input fallbacks.

Would maintainers prefer one of these approaches, or a small feasibility spike comparing UIA observation + one semantic action + WGC capture + packaged startup before choosing? Are there existing Go CLI/MCP compatibility requirements or distribution constraints that should decide this?

References: UIA threading requirements, WinRT APIs in .NET desktop apps, .NET publishing options.

中文说明

结合上面先交付受限 Windows 语义操作预览版的建议，想确认维护者对原生 helper 的实现语言是否已有倾向：保留 Go 并替换逐次调用的 PowerShell 桥接，还是由现有 TypeScript Host 直接监管 C#/.NET helper？

当前 apps/OpenComputerUseWindows 原型是 Go + PowerShell，而 Issue 正文建议采用常驻 C#/.NET 或原生 helper。在开始第一个子任务前，明确这一方向会更有帮助。

两种方案的优缺点如下：

方案	优点	成本与风险
保留 Go，将原生操作迁入常驻 Go helper	可复用现有 Go 入口及相关 CLI/MCP 代码；不增加 .NET 构建工具链；协议与原生执行可集中在一个 helper 中。	UIA COM、WGC/WinRT、回调及对象生命周期需要合适的绑定和明确的所有权管理；必须显式设计 COM 初始化与线程亲和性，不能把 goroutine 等同于所需的执行线程；现有快照和动作语义仍需重构。
TypeScript 直接调用 C#/.NET helper	.NET 有成熟的 Windows SDK/互操作接入路径；显式工作线程、强类型结果和常驻对象注册表适合管理 UIA 与截图会话；无需额外的 Go 消息转发进程。	需要迁移 Windows 执行逻辑，并新增 .NET 构建与发布流程；依赖框架发布要求用户安装运行时，自包含发布则增加包体和随应用维护运行时的成本；COM/WinRT、截图资源释放、provider 卡死与取消仍需专门处理。

保留 Go 不等于保留逐次启动 PowerShell。把 PowerShell 桥接改为常驻也可以作为过渡，但线程、事件、注册表和生命周期管理仍然需要建设。反过来，只有 Go 层确实承担 CLI/MCP 兼容或其他实际职责时，才值得保留 Go → C# 这一层桥接，避免只增加消息转发层。

无论选择哪种语言，都建议保持已讨论的 v1 范围：显式选择一个 HWND，UIA + 目标窗口 WGC 观察，有限语义动作，快照消费，目标重新校验，明确的结果类型，以及 provider 卡死后可重启的受监管子进程。语言选择不应要求 Windows 实现字面意义上的 macOS maka.cu/2 协议，也不应引入前台/全局输入 fallback。

维护者是否倾向其中一种方案，还是希望先通过一个小型可行性验证，对比 UIA 观察 + 一个语义动作 + WGC 截图 + 打包启动 后再决定？是否存在必须保留的 Go CLI/MCP 兼容需求，或会影响选择的分发约束？

参考资料见上方微软官方文档链接。

@sunheyi6

@sunheyi6 asked whether maintainers prefer retaining Go for a long-lived native helper or supervising a C#/.NET helper directly from TypeScript, and whether a feasibility spike should come first.

@sunheyi6 Thanks for laying out the trade-offs so clearly. For the bounded v1 preview, my current preference is a C#/.NET helper supervised directly by the existing TypeScript host, provided a small feasibility spike confirms packaging and lifecycle behavior on the supported Windows preview.

The spike should stay deliberately narrow: long-lived startup/handshake, one MTA UIA observation, one ValuePattern or supported semantic click, target-window WGC capture via CreateForWindow(HWND), cancellation, and restart after a hung provider. It should also exercise the packaged artifact, not only a development machine.

I would keep the existing Go CLI/MCP surface as the product-facing compatibility boundary where needed, but I do not see a reason to retain a Go-to-.NET forwarding layer if Go has no concrete responsibility beyond forwarding messages. Keeping Go is still reasonable if the spike exposes a distribution or existing integration constraint that materially outweighs the .NET interop benefits.

Regardless of the language, the v1 scope and safety rules in your comment remain unchanged: explicit HWND selection, snapshot-bound opaque tokens, target revalidation, separate capture/action paths, typed outcomes, fail-closed unsupported operations, and no foreground/global-input fallback.

Please go ahead with the feasibility spike as a child issue/PR proposal, including the packaging assumptions and a short decision record from the results.

@sunheyi6

Codex-assisted update, posted on behalf of @sunheyi6.

The requested Windows C#/.NET feasibility prototype is now in draft PR #4346, under child issue #4318. The follow-up implementation plan tracks the remaining feasibility evidence, then the real TypeScript backend/approval/packaging integration and packaged Maka E2E for the bounded semantic preview described above.

The local published artifact passes 34 lifecycle checks and 3 protocol regressions, but clean-machine execution, same-window control replacement, and packaging measurements are still open. Neither the spike nor this update claims that Windows Computer Use is already usable in Maka or closes this epic. The user-facing completion criterion is operation from a Maka conversation without manually launching the helper/test driver.

中文说明

此更新由 Codex 协助、代表 @sunheyi6 发布。

维护者要求的 Windows C#/.NET 实验已提交为 Draft PR #4346，对应子 issue #4318。后续实施方案已列出补齐实验验收、正式 TypeScript 后端与授权/打包接入、以及真实 Maka 安装包端到端验证的顺序。

本机发布产物通过 34 项生命周期和 3 项协议回归，但干净机器、同窗口控件替换及打包测量仍未完成。目前不是 Maka 内可用的 Windows 功能，也不关闭本 epic。最终验收标准是用户直接从 Maka 对话操作支持的指定应用，无需手动启动 helper 或测试脚本。

@sunheyi6

Codex-assisted design update, posted on behalf of @sunheyi6 after reviewing #4668 and maka-agent/maka-cu#8 against the intended product boundary and the background-operation patterns used by Open Codex Computer Use, CUA Driver, and Pi Computer Use.

Product boundary: background-only desktop automation

The primary Windows goal should be stated more narrowly and more strongly:

Windows Computer Use automates supported desktop applications in the background without taking over the user's foreground keyboard, mouse, clipboard, or active window.

Browser automation is out of scope here. This does not mean that Maka will not support browser tasks. Those tasks already have a better execution layer in Browser Use/OpenCLI, which can use browser-native structure and commands rather than treating a web page as arbitrary desktop pixels and input devices. Keeping the two responsibilities separate matters because:

Browser Use/OpenCLI can address pages, DOM/accessibility state, tabs, navigation, and browser actions directly, giving more stable targeting and verification than desktop coordinates or synthetic keys.
It can preserve background operation without borrowing the user's global keyboard, pointer, focus, or clipboard.
Reimplementing browser control in the desktop executor would duplicate capability discovery, permissions, retries, observation, and E2E coverage.
Browser compatibility pressure would encourage coordinate clicks, global keyboard input, and foreground activation fallbacks—the exact mechanisms excluded by the strict non-interference contract.
A clear routing rule is easier for the agent and user to understand: web content goes through Browser Use/OpenCLI; native desktop UI goes through Windows Computer Use; unsupported hybrid surfaces fail explicitly or use a separately reviewed adapter.

The exclusion therefore improves both browser reliability and desktop safety. It is an ownership boundary, not a loss of product capability.

This boundary intentionally trades breadth for non-interference. Some desktop applications and controls cannot be operated safely in the background. In those cases the correct result is a typed unsupported or refused outcome, not a foreground fallback.

Required invariant

For every action, the executor must record and verify the foreground HWND/PID and pointer position before and after dispatch. A successful background action must not intentionally:

call SetForegroundWindow, SetFocus, or equivalent activation APIs;
use global SendInput for keyboard or pointer input;
move the physical cursor;
replace or temporarily use the user's clipboard;
send keystrokes to whichever window happens to be foreground;
automatically fall back from a semantic/background path to a foreground/global path.

If the target application activates itself as a side effect, the executor should detect the foreground change, stop further actions, and return a typed interference result. Restoring the previous foreground window may be a best-effort mitigation, but it must not convert the action into a successful non-interfering operation.

Revised delivery plan

Phase 1 — safe semantic preview

Keep the existing shared Maka host/service boundary and snapshot lifecycle. Limit the Windows executor to already-running, explicitly selected desktop windows and background-safe semantic operations:

observation through UI Automation;
target-window capture through Windows Graphics Capture;
click_element only through supported semantic patterns such as Invoke, Toggle, or SelectionItem;
set_value through ValuePattern.SetValue;
read-only tree, text, window, and application queries;
explicit unsupported/refused results for keyboard, point, drag, generic launch, or any action requiring foreground ownership.

The handshake must advertise only capabilities that meet this contract. In particular, keyboard actions should not be advertised until they have a background-safe implementation and reliable verification.

Phase 2 — DPI and coordinate correctness

Declare Per-Monitor-V2 DPI awareness, use per-window DPI, and keep logical UIA coordinates separate from physical capture coordinates. Test 100%, 125%, 150%, and 200% scaling, including mixed-DPI and negative-coordinate monitor layouts. Coordinate-based actuation remains disabled unless a background-safe path is proven.

Phase 3 — verified background text/input adapters

Add text support as an explicit adapter ladder, never as an automatic global fallback:

UIA ValuePattern.SetValue with readback;
known control-specific background APIs/messages for supported Edit/RichEdit-style controls, with post-action verification;
application-specific accessibility or automation APIs;
otherwise unsupported.

SetForegroundWindow, SetFocus, global SendInput, clipboard substitution, and physical-pointer movement remain outside the background-only contract.

Phase 4 — release qualification

distributionReady must stay false until all release gates are mechanically enforced for the exact artifact:

missing helper fails packaging when Windows distribution is enabled;
exact file set, sizes, and SHA-256 digests are verified in the packaged application;
the build is locked and uses an explicit Windows target;
the binary is either truly self-contained/static where intended, or every runtime dependency is shipped and verified;
Authenticode is actually verified rather than represented only by provenance metadata;
clean-machine packaged conversation E2E passes.

Acceptance evidence

The decisive E2E should keep a user continuously typing in a foreground application while Maka manipulates a different background desktop application. The test must prove:

foreground HWND/PID and physical pointer position remain unchanged;
all user keystrokes stay in the foreground application;
the background action is verified, or it fails with a typed refusal/unsupported result;
no retry can duplicate an action whose outcome is unknown;
observation and semantic actions work under occlusion and the declared minimized-window policy;
the same guarantees hold across the supported DPI matrix;
the packaged artifact works on a clean Windows 11 machine.

This is a deliberate product constraint, not a temporary implementation detail. References that support useful techniques—explicit focus policy and restoration, focus-preserving actions, background/foreground mode separation, and Per-Monitor-V2 handling—should inform the implementation, but Maka's success criterion is stricter: no successful action may interfere with the user's active desktop work.

Ablation conclusion

The minimal architecture remains one shared MakaCuService/host boundary plus one Windows native helper. A second public protocol, a browser-specific path, an automatic foreground mode, and a compatibility/global-input subsystem are unnecessary for this goal and should not be added. Removing those paths makes the supported capability set smaller, but preserves the core background-only guarantee and makes failures honest and testable.

中文说明

这是一条由 Codex 协助、代表 @sunheyi6 发布的设计更新。更新依据是对 #4668 和 maka-agent/maka-cu#8 的审查，并参考了 Open Codex Computer Use、CUA Driver、Pi Computer Use 中与后台操作相关的做法。

产品边界：只做后台桌面自动化

Windows Computer Use 的首要目标应更明确地定义为：

在后台操作受支持的桌面应用，不抢占用户正在使用的前台窗口、键盘、鼠标和剪贴板。

浏览器不属于这里的范围，但这不表示 Maka 不支持浏览器任务。浏览器任务已经有更合适的 Browser Use/OpenCLI：它可以直接使用浏览器自身的页面结构和命令，而不必把网页当成一块桌面像素，再去模拟人的键鼠。两者分工必须清楚，原因是：

Browser Use/OpenCLI 能直接定位页面、DOM/无障碍状态、标签页、导航和浏览器动作，比桌面坐标或模拟按键更稳定，也更容易验证结果；
它可以在不占用用户全局键盘、鼠标、焦点和剪贴板的情况下保持后台操作；
如果桌面执行器再次实现浏览器控制，会重复建设能力发现、权限、重试、观察和 E2E 测试；
为了兼容各种网页，很容易被迫加入坐标点击、全局键盘和抢前台 fallback，而这些正是严格不干扰合同明确排除的机制；
清晰的路由规则也更容易理解：网页内容走 Browser Use/OpenCLI，原生桌面 UI 走 Windows Computer Use；混合界面如果没有经过单独验证的适配器，就明确返回不支持。

所以排除浏览器是职责划分，不是产品能力缺失；它同时提高浏览器操作的可靠性和桌面操作的安全性。

严格后台必然会减少可操作范围。有些桌面应用或控件无法安全地在后台操作，此时正确行为是返回明确的 unsupported 或 refused，而不是偷偷切到前台执行。

必须遵守的不变量

每次动作前后都要记录并校验前台 HWND/PID 和鼠标位置。一次被判定为成功的后台动作不得主动：

调用 SetForegroundWindow、SetFocus 或同类激活接口；
使用全局 SendInput 输入键盘或鼠标事件；
移动物理鼠标；
替换或临时占用用户剪贴板；
把按键发给当时恰好处于前台的窗口；
从语义/后台路径自动降级到前台/全局输入路径。

如果目标应用因为自身行为抢到前台，执行器应检测到前台变化、停止后续动作，并返回明确的干扰结果。恢复原前台窗口只能作为尽力补救，不能把这次操作算成“未干扰用户”的成功。

调整后的交付计划

第一阶段：安全的语义操作预览版

保留现有共享 Maka Host/Service 边界和 snapshot 生命周期。Windows 执行器只针对用户明确选择、已经运行的桌面窗口，并仅开放可以保证后台安全的语义操作：

通过 UI Automation 观察界面；
通过 Windows Graphics Capture 截取目标窗口；
只有元素支持 Invoke、Toggle、SelectionItem 等语义模式时才执行 click_element；
通过 ValuePattern.SetValue 执行 set_value；
只读的控件树、文本、窗口和应用查询；
键盘、坐标点击、拖拽、通用启动以及任何需要前台所有权的动作，统一返回明确的 unsupported/refused。

握手只能声明真正满足以上约束的能力。在具备安全的后台实现和可靠验证前，不应声明 keyboard 能力。

第二阶段：DPI 与坐标正确性

启用 Per-Monitor-V2 DPI awareness，使用每窗口 DPI，并明确区分 UIA 逻辑坐标和截图物理坐标。覆盖 100%、125%、150%、200% 缩放、混合 DPI 以及负坐标多屏布局。在证明存在后台安全路径前，仍不开启基于坐标的执行。

第三阶段：可验证的后台文本/输入适配器

文本输入按明确的适配阶梯实现，绝不自动降级到全局输入：

UIA ValuePattern.SetValue，并做结果回读；
对已知 Edit/RichEdit 类控件使用受支持的后台接口或消息，并验证操作结果；
使用应用自身的无障碍或自动化 API；
其他情况返回 unsupported。

SetForegroundWindow、SetFocus、全局 SendInput、剪贴板替换和物理鼠标移动始终不属于严格后台合同。

第四阶段：发布资格

只有同一个确切构建产物满足以下可机械验证的门槛后，distributionReady 才能改为 true：

开启 Windows 分发时，helper 缺失必须导致打包失败；
安装包内对准确文件集合、大小和 SHA-256 做校验；
构建使用锁定依赖和明确的 Windows target；
二进制要么确实自包含/静态链接，要么随包携带并验证全部运行时依赖；
真正验证 Authenticode，不能只相信 provenance 文本字段；
在干净 Windows 11 机器上通过安装包真实会话 E2E。

验收证据

最关键的 E2E 是：用户一直在前台应用中打字，同时 Maka 操作另一个后台桌面应用。测试必须证明：

前台 HWND/PID 和物理鼠标位置不变；
用户所有按键仍只进入前台应用；
后台动作得到验证，或者返回明确的拒绝/不支持；
对结果未知的动作不会因为重试而重复执行；
遮挡状态和声明支持的最小化策略下，观察与语义操作行为正确；
支持的 DPI 组合下仍满足上述保证；
打包产物可在干净 Windows 11 机器运行。

这是明确的产品约束，不是临时实现细节。Open Codex、CUA Driver、Pi Computer Use 中的显式焦点策略、焦点恢复、前后台模式区分、保留焦点动作和 Per-Monitor-V2 处理可以作为技术参考，但 Maka 的成功标准更严格：任何被报告为成功的动作都不能干扰用户当前的桌面操作。

消融结论

满足目标所需的最小架构仍然是一套共享 MakaCuService/Host 边界，加一个 Windows 原生 helper。不需要第二套公开协议、浏览器专用路径、自动前台模式或兼容性全局输入子系统。移除这些路径会缩小功能覆盖面，但能保住严格后台保证，并让失败结果真实、明确、可测试。

Implementation update: the strict background-only contract is now pushed to both PR branches.

maka-agent/maka-cu#8 now advertises and accepts only semantic click and set_value, removes foreground/global keyboard and process-launch paths, rejects foreground targets, monitors foreground/pointer/clipboard interference, initializes Per-Monitor-V2 DPI awareness, and builds with an explicit static-CRT Windows target.
apache/maka#4668 now fails packaging closed, removes the caller-authored provenance readiness escape hatch, verifies the exact packaged helper file manifest, and requires valid Authenticode when readiness is enabled.
Browser automation remains excluded and is routed to Browser Use/OpenCLI because that layer owns DOM/accessibility, tabs, navigation, page lifecycle, and browser command state without desktop-global fallbacks.

Automated validation passed for the new code paths. distributionReady intentionally remains false until the same immutable signed artifact passes clean-machine, concurrent-user/mixed-DPI, and packaged real-conversation E2E.

中文说明

执行进度：严格后台操作契约已经推送到两个 PR 分支。

maka-agent/maka-cu#8 现在只声明并接受语义 click 和 set_value，删除前台/全局键盘与进程启动路径，拒绝前台目标，监测前台/指针/剪贴板干扰，启用 Per-Monitor-V2 DPI 感知，并使用明确的静态 CRT Windows target 构建。
apache/maka#4668 现在默认失败关闭打包，删除调用方自行填写 provenance 即可开启 readiness 的逃生口，验证打包 helper 的精确文件清单，并在 readiness 开启时强制要求有效 Authenticode。
浏览器自动化继续排除，并路由到 Browser Use/OpenCLI，因为该层直接掌握 DOM/可访问性树、标签页、导航、页面生命周期和浏览器命令状态，不需要桌面全局输入兜底。

新增代码路径的自动化验证已经通过。distributionReady 有意保持 false，直到同一份不可变且已签名的 artifact 通过 clean-machine、并发用户/混合 DPI 和 packaged real-conversation E2E。

Two Windows-specific items worth pinning down in the executor contract before the implementation stabilises:1. Coordinate space, stated per capture. UIA element bounds are window-relative and DPI-dependent, while a capture API usually returns screenshot pixels for one monitor. Mixing the two silently is the most common source of wrong clicks on Windows - and it differs by a constant if you compare GetWindowRect (which includes the invisible resize border) with DwmGetWindowAttribute(DWMWA_EXTENDED_FRAME_BOUNDS) (the visible frame). Making the contract "the capture reports its origin and scale, otherwise mapping fails closed" removes that whole class.2. Secure desktop and session state. On the secure desktop (UAC), and when the session is locked or disconnected over RDP, UIA returns incomplete trees and injected input is dropped rather than failing loudly. Treating those as explicit "cannot act now" states is much cheaper than retrying into them.On evidence: a multi-monitor case with non-100% scaling, plus a locked-session case, would cover more real failures than another single-window flow.

feat(windows): harden and productionize the maka-cu Computer Use executor #3785

Description

Relationship

Context

Review Findings

1. Observation/action binding is too weak

2. The screenshot path can disagree with the UIA tree

3. Win32 input currently reports success without verification

4. The wheel fallback uses the wrong coordinate space

5. DPI, threading, and per-call process startup need a production decision

Proposed Direction

A. Reuse the platform-neutral host contract

B. Use stable target identity, not app-name identity

C. Make capture and input capability-driven

D. Prefer a long-lived native Windows bridge

E. Define the Windows security/session contract

Proposed Deliverables

Acceptance Criteria

Protocol and safety

Observation and capture

Actions

Release evidence

Non-goals

References

Activity

M4n5ter commented on Aug 25, 2026

sunheyi6 commented on Aug 31, 2026

liugddx commented on Aug 31, 2026

sunheyi6 commented on Aug 31, 2026

sunheyi6 commented on Sep 4, 2026

Product boundary: background-only desktop automation

Required invariant

Revised delivery plan

Phase 1 — safe semantic preview

Phase 2 — DPI and coordinate correctness

Phase 3 — verified background text/input adapters

Phase 4 — release qualification

Acceptance evidence

Ablation conclusion

产品边界：只做后台桌面自动化

必须遵守的不变量

调整后的交付计划

第一阶段：安全的语义操作预览版

第二阶段：DPI 与坐标正确性

第三阶段：可验证的后台文本/输入适配器

第四阶段：发布资格

验收证据

消融结论

sunheyi6 commented on Sep 4, 2026

youlemei2022-web commented on Sep 21, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions