HandsFree

HandsFree is a free macOS assistant that uses OpenAI Realtime and MCPs to interact with your computer.

The core idea is simple: speech and text go through OpenAI Realtime, and actions happen through tools. MCPs are the first-class way to add those tools. If an app exposes an MCP server, HandsFree can discover its tools and use them directly instead of guessing from pixels.

Why

Realtime voice models are finally fast enough to feel useful at the computer. HandsFree keeps that loop small: listen, understand the request, call the right tool, and report back.

The goal is not to push every bit of intelligence into the realtime model. The realtime model should do the minimum necessary orchestration while MCP servers provide precise, app-specific control.

MCP-First

MCPs are treated as regular product surface, not an advanced escape hatch. Add MCP servers in settings.json, restart or refresh HandsFree, and they appear as available computer capabilities.

We know MCPs are not widely used for UI interaction yet. We think the standard is already adequate for overtaking UI controls when apps expose useful actions, state, and navigation as tools. That makes control faster, more reliable, and reusable by other assistants or automation systems.

Chrome DevTools MCP example:

{
  "mcpServers": {
    "chrome-devtools": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "chrome-devtools-mcp@latest"]
    }
  }
}

OpenWork UI MCP example:

{
  "mcpServers": {
    "openwork-ui": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "openwork-ui-mcp"],
      "env": {
        "OPENWORK_UI_CONTROL_DISCOVERY": "~/Library/Application Support/com.differentai.openwork/openwork-ui-control.json"
      }
    },
    "chrome-devtools": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "chrome-devtools-mcp@latest"]
    }
  }
}

The OPENWORK_UI_CONTROL_DISCOVERY env tells openwork-ui-mcp where the running OpenWork app writes its bridge file. Without it the MCP process may not find the app. For dev builds use com.differentai.openwork.dev instead of com.differentai.openwork.

Computer Use

Computer Use is the one special internal subsystem. It is not a normal user-added MCP because it combines two pieces that have to work together:

• The Electron side runs the GPT-5.5 computer-use loop, sends screenshots to the model, receives computer actions, and manages cancellation/progress.
• The Swift Computer Use helper owns macOS Accessibility and Screen Recording permissions, captures screenshots, and executes native mouse, keyboard, clipboard, app, and grid tools.

Electron does not request Accessibility or Screen Recording itself. When the user explicitly chooses to use Computer Use, Electron asks the Swift helper to request the missing permission, restarts that helper, and polls it until the helper reports that access is valid.

Use MCPs first when an app has a good tool interface. Use Computer Use when HandsFree needs to interact with the Mac itself or with UI that has not exposed an MCP server yet.

Building Your Own MCP

Want to add first-class voice control for your app? See:

• Integration Guide — Best practices, design principles, and step-by-step instructions for building an MCP that works well with HandsFree.
• Starter Template — Copy this directory to get a working MCP server with example tools in under 5 minutes.
• Marketplace — Submit a PR to list your MCP in HandsFree's built-in connector marketplace.

Development

pnpm install
pnpm run dev:ui
pnpm run dev

Build

pnpm run build:ui
pnpm run package:dir

Generated output is intentionally ignored under dist/ and dist-electron/.