Discovery (from #11506 audit)
Two parallel, both-wired desktop-control stacks exist, driving different X displays — so an agent using one path acts on a screen the human observer can't see.
Stack A — pyautogui / GUIController
autobot-backend/worker_node.py:58 imports GUIController; instantiated at :90 as self.gui_controller
autobot-backend/task_handlers/gui_handlers.py:53 calls ctx.worker.gui_controller.read_text_from_region(...)
- Drives Xvfb
:99 — autobot-backend/gui_controller.py:58 (Xvfb :99 ...), config.display = ":99" at :62
Stack B — xdotool / vnc_manager (agent MCP tools + human observation)
autobot-backend/api/vnc_manager.py xdotool click/type/etc.; exposed to agents via autobot-backend/api/vnc_mcp.py
- Human observes this via x11vnc/noVNC — runs on display
:1 (vnc_manager.py:115 "Start VNC server on display :1")
Why this matters
Proposed direction
Dependency
Blocks / must be resolved as part of #11506 T1 (control-lock) — the lock and the display must be the same for both the agent and the human.
Evidence-based; file:line cited above.
Discovery (from #11506 audit)
Two parallel, both-wired desktop-control stacks exist, driving different X displays — so an agent using one path acts on a screen the human observer can't see.
Stack A — pyautogui / GUIController
autobot-backend/worker_node.py:58importsGUIController; instantiated at:90asself.gui_controllerautobot-backend/task_handlers/gui_handlers.py:53callsctx.worker.gui_controller.read_text_from_region(...):99—autobot-backend/gui_controller.py:58(Xvfb :99 ...),config.display = ":99"at:62Stack B — xdotool / vnc_manager (agent MCP tools + human observation)
autobot-backend/api/vnc_manager.pyxdotool click/type/etc.; exposed to agents viaautobot-backend/api/vnc_mcp.py:1(vnc_manager.py:115"Start VNC server on display :1")Why this matters
:99(Stack A) is invisible to a human watching:1(Stack B). Breaks the "observe + take over" goal of Umbrella: agent-operated desktop sessions with safe human takeover (x11vnc) #11506.pyautoguivsxdotool) with divergent humanization, capture, and audit paths.Proposed direction
gui_handlersonto the canonical path (read_text_from_regionhas no xdotool-path equivalent yet; that capability must be preserved).Dependency
Blocks / must be resolved as part of #11506 T1 (control-lock) — the lock and the display must be the same for both the agent and the human.
Evidence-based; file:line cited above.