Skip to content

Two parallel desktop-control stacks on different displays (:99 vs :1) — consolidate to canonical path #11579

Description

@mrveiss

Discovery (from #11506 audit)

Two parallel, both-wired desktop-control stacks exist, driving different X displays — so an agent using one path acts on a screen the human observer can't see.

Stack A — pyautogui / GUIController

  • autobot-backend/worker_node.py:58 imports GUIController; instantiated at :90 as self.gui_controller
  • autobot-backend/task_handlers/gui_handlers.py:53 calls ctx.worker.gui_controller.read_text_from_region(...)
  • Drives Xvfb :99 — autobot-backend/gui_controller.py:58 (Xvfb :99 ...), config.display = ":99" at :62

Stack B — xdotool / vnc_manager (agent MCP tools + human observation)

  • autobot-backend/api/vnc_manager.py xdotool click/type/etc.; exposed to agents via autobot-backend/api/vnc_mcp.py
  • Human observes this via x11vnc/noVNC — runs on display :1 (vnc_manager.py:115 "Start VNC server on display :1")

Why this matters

Proposed direction

Dependency

Blocks / must be resolved as part of #11506 T1 (control-lock) — the lock and the display must be the same for both the agent and the human.

Evidence-based; file:line cited above.

Activity

  1. mrveiss commented on Jul 22, 2026

    @mrveiss
    OwnerAuthor

    Closed — merged to Dev_new_gui via #11973 (squash f546e38). Both desktop-control stacks now converge on the human-observed display via single source of truth NetworkConstants.DESKTOP_DISPLAY (env AUTOBOT_DESKTOP_DISPLAY, default :1); GUIController attaches to it instead of spawning Xvfb :99; read_text_from_region (OCR) ported onto the real controller (was only on the dummy) so no capability lost; all 11 hardcoded :1 in vnc_manager/vnc_mcp use the constant; gui_handlers async calls now awaited; pyautogui.size() race guarded. 26 tests pass. Fast-follows filed: #11970 (inverted platform gate — real controller doesn't run in prod), #11971 (worker_node_test pre-existing errors). #11506 T1 control-lock hook point documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions