Skip to content

worker_node.py platform gate for GUIController is inverted (dummy on Linux, real pyautogui/Xvfb on non-Linux) #11970

Description

@mrveiss

Discovery (from #11579 display-convergence work)

autobot-backend/worker_node.py:52-60:

if sys.platform.startswith("linux"):
    from gui_controller_dummy import GUIController
    GUI_AUTOMATION_SUPPORTED = False
else:
    from gui_controller import GUIController
    GUI_AUTOMATION_SUPPORTED = True

This is backwards for AutoBot's deployment topology:

Net effect: Stack A's real desktop-automation code path (gui_controller.py) currently never executes on the platform it's deployed to. The gui_read_text_from_region / gui_click_element / gui_move_mouse / gui_bring_window_to_front task handlers (autobot-backend/task_handlers/gui_handlers.py) always resolve to the dummy's skip-stubs in production.

Why not fixed in #11579

Flipping the platform gate is NOT a simple swap:

  1. Not every worker node that constructs WorkerNode() (and therefore GUIController()) is guaranteed to have a live desktop/VNC display — some worker VMs are headless task workers with no X server at all. pyautogui.size() (called at GUIController.__init__) requires a live X connection; flipping the gate unconditionally could break worker startup on non-desktop Linux nodes.
  2. The real gui_controller.py is missing several methods that task_handlers/gui_handlers.py already calls unconditionally: click_element, move_mouse, bring_window_to_front (only gui_controller_dummy.py implements these today — read_text_from_region was ported to the real controller in Two parallel desktop-control stacks on different displays (:99 vs :1) — consolidate to canonical path #11579, but the other three were left as a deliberate scope boundary).
  3. Separately, tools/tool_registry.py never creates a gui_read_text_from_region or gui_move_mouse task at all (gui_click_element/gui_type_text/gui_bring_window_to_front are the only ones wired from the agent side) — so parts of this task-type surface may already be dead/unreachable code, independent of the platform-gate bug.

Proposed direction

  • Gate GUIController selection on desktop-display availability (e.g., "is the canonical NetworkConstants.DESKTOP_DISPLAY VNC/X server reachable on this node?"), not on sys.platform.
  • Add the missing click_element / move_mouse / bring_window_to_front methods to the real gui_controller.py for interface parity with gui_controller_dummy.py.
  • Audit tools/tool_registry.py for the missing gui_read_text_from_region / gui_move_mouse task-creation wiring (or confirm intentionally unreachable and remove/document).

Evidence: autobot-backend/worker_node.py:52-60, autobot-backend/gui_controller.py, autobot-backend/gui_controller_dummy.py, autobot-backend/task_handlers/gui_handlers.py, autobot-backend/tools/tool_registry.py:508-537.

Related: #11579 (display convergence, this PR), #11506 (desktop-control umbrella).

Activity

  1. mrveiss commented on Jul 22, 2026

    @mrveiss
    OwnerAuthor

    Additional discovery while fixing the missing-`await` bug in `task_handlers/gui_handlers.py` (#11579 review follow-up): `GUITypeTextHandler` calls `gui_controller.type_text(text, interval)` (2 positional args), but the real `gui_controller.GUIController.type_text(self, text)` only accepts `text` — no `interval` parameter. Only `gui_controller_dummy.GUIController.type_text(self, text, interval=0.0)` (the dummy, currently the ACTIVE controller on Linux/prod per this issue) has the matching signature. This is the same interface-parity gap as `click_element`/`move_mouse`/`bring_window_to_front` and should be closed together when this issue is worked.

  2. mrveiss commented on Jul 22, 2026

    @mrveiss
    OwnerAuthor

    Closed — merged via PR #12071 (c7290bfb2, real CI green). Fixed the inverted GUIController platform gate (Linux → real controller, with a try/except dummy fallback for headless Linux/CI) + added the missing gui_controller methods the handler calls. 37 tests pass.

  3. mrveiss commented on Jul 22, 2026

    @mrveiss
    OwnerAuthor

    Closed — merged to Dev_new_gui via #12071 (squash commit c7290bfb2). Platform gate flip verified correct: sys.platform.startswith('linux') → real GUIController (AutoBot's Xvfb/pyautogui stack is Linux-only), non-Linux → dummy, with a graceful headless-Linux try/except fallback. gui_controller parity methods added (click_element/move_mouse/bring_window_to_front/type_text interval). Code review: APPROVE-WITH-NITS; all 33 CI checks green. Discoveries: sibling connection_utils get_nested pattern already tracked as #12069.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions