You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
worker_node.py platform gate for GUIController is inverted (dummy on Linux, real pyautogui/Xvfb on non-Linux) #11970
Today, Linux (production) uses gui_controller_dummy.GUIController — a total no-op (read_text_from_region/click_element/etc. all return {"status": "success", ...skipped} without touching any display).
Net effect: Stack A's real desktop-automation code path (gui_controller.py) currently never executes on the platform it's deployed to. The gui_read_text_from_region / gui_click_element / gui_move_mouse / gui_bring_window_to_front task handlers (autobot-backend/task_handlers/gui_handlers.py) always resolve to the dummy's skip-stubs in production.
Not every worker node that constructs WorkerNode() (and therefore GUIController()) is guaranteed to have a live desktop/VNC display — some worker VMs are headless task workers with no X server at all. pyautogui.size() (called at GUIController.__init__) requires a live X connection; flipping the gate unconditionally could break worker startup on non-desktop Linux nodes.
The real gui_controller.py is missing several methods that task_handlers/gui_handlers.py already calls unconditionally: click_element, move_mouse, bring_window_to_front (only gui_controller_dummy.py implements these today — read_text_from_region was ported to the real controller in Two parallel desktop-control stacks on different displays (:99 vs :1) — consolidate to canonical path #11579, but the other three were left as a deliberate scope boundary).
Separately, tools/tool_registry.py never creates a gui_read_text_from_region or gui_move_mouse task at all (gui_click_element/gui_type_text/gui_bring_window_to_front are the only ones wired from the agent side) — so parts of this task-type surface may already be dead/unreachable code, independent of the platform-gate bug.
Proposed direction
Gate GUIController selection on desktop-display availability (e.g., "is the canonical NetworkConstants.DESKTOP_DISPLAY VNC/X server reachable on this node?"), not on sys.platform.
Add the missing click_element / move_mouse / bring_window_to_front methods to the real gui_controller.py for interface parity with gui_controller_dummy.py.
Audit tools/tool_registry.py for the missing gui_read_text_from_region / gui_move_mouse task-creation wiring (or confirm intentionally unreachable and remove/document).
Additional discovery while fixing the missing-`await` bug in `task_handlers/gui_handlers.py` (#11579 review follow-up): `GUITypeTextHandler` calls `gui_controller.type_text(text, interval)` (2 positional args), but the real `gui_controller.GUIController.type_text(self, text)` only accepts `text` — no `interval` parameter. Only `gui_controller_dummy.GUIController.type_text(self, text, interval=0.0)` (the dummy, currently the ACTIVE controller on Linux/prod per this issue) has the matching signature. This is the same interface-parity gap as `click_element`/`move_mouse`/`bring_window_to_front` and should be closed together when this issue is worked.
Closed — merged via PR #12071 (c7290bfb2, real CI green). Fixed the inverted GUIController platform gate (Linux → real controller, with a try/except dummy fallback for headless Linux/CI) + added the missing gui_controller methods the handler calls. 37 tests pass.
Discovery (from #11579 display-convergence work)
autobot-backend/worker_node.py:52-60:This is backwards for AutoBot's deployment topology:
api/vnc_manager.py, Xtigervnc:1) that Two parallel desktop-control stacks on different displays (:99 vs :1) — consolidate to canonical path #11579 convergesgui_controller.pyonto only exists on Linux (Xvfb/Xtigervnc are Linux/Unix-only tools).gui_controller_dummy.GUIController— a total no-op (read_text_from_region/click_element/etc. all return{"status": "success", ...skipped}without touching any display).gui_controller.GUIController— which after Two parallel desktop-control stacks on different displays (:99 vs :1) — consolidate to canonical path #11579 targets the canonical display viaNetworkConstants.DESKTOP_DISPLAY, but non-Linux dev machines don't runapi.vnc_manager's Xtigervnc server at all, and pyautogui/Xvfb-adjacent tooling isn't Windows/macOS-native.Net effect: Stack A's real desktop-automation code path (
gui_controller.py) currently never executes on the platform it's deployed to. Thegui_read_text_from_region/gui_click_element/gui_move_mouse/gui_bring_window_to_fronttask handlers (autobot-backend/task_handlers/gui_handlers.py) always resolve to the dummy's skip-stubs in production.Why not fixed in #11579
Flipping the platform gate is NOT a simple swap:
WorkerNode()(and thereforeGUIController()) is guaranteed to have a live desktop/VNC display — some worker VMs are headless task workers with no X server at all.pyautogui.size()(called atGUIController.__init__) requires a live X connection; flipping the gate unconditionally could break worker startup on non-desktop Linux nodes.gui_controller.pyis missing several methods thattask_handlers/gui_handlers.pyalready calls unconditionally:click_element,move_mouse,bring_window_to_front(onlygui_controller_dummy.pyimplements these today —read_text_from_regionwas ported to the real controller in Two parallel desktop-control stacks on different displays (:99 vs :1) — consolidate to canonical path #11579, but the other three were left as a deliberate scope boundary).tools/tool_registry.pynever creates agui_read_text_from_regionorgui_move_mousetask at all (gui_click_element/gui_type_text/gui_bring_window_to_frontare the only ones wired from the agent side) — so parts of this task-type surface may already be dead/unreachable code, independent of the platform-gate bug.Proposed direction
GUIControllerselection on desktop-display availability (e.g., "is the canonicalNetworkConstants.DESKTOP_DISPLAYVNC/X server reachable on this node?"), not onsys.platform.click_element/move_mouse/bring_window_to_frontmethods to the realgui_controller.pyfor interface parity withgui_controller_dummy.py.tools/tool_registry.pyfor the missinggui_read_text_from_region/gui_move_mousetask-creation wiring (or confirm intentionally unreachable and remove/document).Evidence:
autobot-backend/worker_node.py:52-60,autobot-backend/gui_controller.py,autobot-backend/gui_controller_dummy.py,autobot-backend/task_handlers/gui_handlers.py,autobot-backend/tools/tool_registry.py:508-537.Related: #11579 (display convergence, this PR), #11506 (desktop-control umbrella).