Umbrella: agent-operated desktop sessions with safe human takeover (x11vnc) #11506

Goal

Let an agent operate a real graphical desktop session (genuine clicks/typing, not screenshot-only), while a human can observe the same live session and take over safely. Provisioned per-role by SLM.

Key finding — most of this already exists (reuse, do not rebuild)

A three-part codebase audit confirmed the actuation, perception, observation, and provisioning stacks are already wired in:

• Real actuation (xdotool): mouse click/drag/scroll + keyboard type/special-keys with humanized timing — autobot-backend/api/vnc_manager.py
• Agent already drives it (MCP tools): desktop_mouse_click, desktop_keyboard_type, desktop_special_key, desktop_screenshot, desktop_observe_state — autobot-backend/api/vnc_mcp.py
• Perception: screenshot → CLIP/BLIP-2 → UI-element detection → cross-modal semantic search — autobot-backend/computer_vision/ + autobot-backend/multimodal_processor/; OCR + cv2.matchTemplate + wait_for_text/wait_for_image in vnc_manager.py
• Observation + provisioning: VNC role (tigervnc, x11vnc, websockify, novnc, xfce4) and xrdp role (xrdp + xorgxrdp, 3389), both gated per-node in autobot-slm-backend/ansible/playbooks/provision-fleet-roles.yml (Phase 4a); encrypted credential auto-registration into NodeCredential
• Audit logging: autobot-backend/integrations/desktop_tracking.py

Architecture decision

Keep x11vnc for agent-driven + human-observable roles. x11vnc attaches to the display the agent is already driving, so the human sees the same pixels and can take over. xrdp's xorgxrdp spawns a fresh session on connect (hides the agent's work) — reserve xrdp for "human logs into a Linux desktop by hand, no agent" roles. This is expressed by a per-role remote_access flag (T2).

Confirmed gaps (the actual work)

The audits found exactly three gaps. T1 is the only safety-critical one.

Tasks

• T1: Agent↔human control-lock (input arbitration) → PR #____
  • No arbitration exists today; agent + human would both feed the same X input queue on takeover.
  • Subtask 1.1: human_active control-lock state (per-session), owner + release semantics — new module in autobot-backend/api/
  • Subtask 1.2: Gate all actuation in vnc_manager.py (click/type/scroll/drag/special-key) on the lock — mute agent input while human is active
  • Subtask 1.3: Signal takeover/handback from the VNC observe path (vnc_proxy.py) and expose lock state via MCP (vnc_mcp.py) so the agent knows to pause
  • Subtask 1.4: Frontend affordance — "Take control / Release" toggle on the desktop-stream view; surface lock owner
• T2: Per-role remote-access wiring → PR #____
  • Subtask 2.1: Add ManifestRemoteAccess (enabled, type: none|xrdp|vnc, display config) field to RoleManifest — autobot-slm-backend/models/manifest.py
  • Subtask 2.2: Add xrdp to ROLE_DEPENDENCIES / ROLE_ANSIBLE_GROUPS — autobot-slm-backend/services/role_registry.py
  • Subtask 2.3: Gate Phase 4a role execution on remote_access.enabled from the manifest; add manifest files for the vnc/xrdp roles
• T3 (optional): AT-SPI desktop-accessibility perception tier → PR #____
  • Today structured perception exists only for browsers (Playwright ARIA); native desktop is screenshot/CV only.
  • Subtask 3.1: pyatspi accessibility-tree reader → structured elements (role/label/coords) as a perception tier above CV
  • Subtask 3.2: Prefer AT-SPI targets over template/OCR when available in the agent's element-resolution path

Out of scope

• Full autonomous VLM→coordinates→click loop (perception currently stops at detection; agent orchestrates via MCP). Revisit separately if full autonomy is wanted.
• Wayland actuation (X11 only — Wayland blocks synthetic input).

Verification

• T1: with the agent actively clicking, a human takes control → agent input stops; release → agent resumes. Audit log shows the handoff.
• T2: a role with remote_access: vnc provisions x11vnc + desktop; remote_access: none provisions neither.
• T3: agent resolves a native-app button via AT-SPI (no screenshot) and clicks its reported coordinates.