Skip to content

Comments

Add Browser Env Integration#732

Merged
willccbb merged 34 commits intoPrimeIntellect-ai:mainfrom
filip-michalsky:fm/add-browser-env
Jan 30, 2026
Merged

Add Browser Env Integration#732
willccbb merged 34 commits intoPrimeIntellect-ai:mainfrom
filip-michalsky:fm/add-browser-env

Conversation

@filip-michalsky
Copy link
Contributor

@filip-michalsky filip-michalsky commented Jan 15, 2026

Description

Adds BrowserEnv - a unified browser automation integration for the verifiers library supporting two operational modes:

DOM Mode (mode="dom")

  • Uses the Stagehand Python SDK for natural language browser control
  • Tools: navigate, observe, act, extract - Stagehand's AI-driven primitives
  • Ideal for tasks that benefit from semantic understanding of page elements

CUA Mode (mode="cua")

  • Vision-based primitives for Computer Use Agent workflows
  • Tools: click, double_click, type_text, keypress, scroll, goto, back, forward, wait, screenshot
  • Automatic sandbox deployment - CUA server is deployed automatically to sandbox containers
  • Three execution modes (fastest to most flexible):
    1. Pre-built image (default): Uses deepdream19/cua-server:latest for ~5-10s startup
    2. Binary upload: Builds and uploads custom server version (~30-60s startup)
    3. Manual server: Connect to locally running CUA server for development

Both modes support local browser execution or Browserbase cloud infrastructure.

What's included:

  • verifiers/envs/integrations/browser_env/ - Core integration (BrowserEnv, DOMMode, CUAMode, CUASandboxMode)
  • assets/templates/browserbase/cua/ - TypeScript CUA server with Docker build/runtime configs
  • environments/browser_dom_example/ - Minimal DOM mode example
  • environments/browser_cua_example/ - Minimal CUA mode example
  • New [browser] extra: uv add 'verifiers[browser]'

Benchmarks (GAIA, WebVoyager, Mind2Web) have been pushed to Prime Hub under the browserbase/ namespace.

Type of Change

  • New feature (non-breaking change which adds functionality)

Testing

# DOM mode
prime eval run browser-dom-example -m openai/gpt-4o-mini

# CUA mode (pre-built image - default, fastest)
prime eval run browser-cua-example -m openai/gpt-4o-mini

# CUA mode (binary upload - custom server)
prime eval run browser-cua-example -m openai/gpt-4o-mini -a '{"use_prebuilt_image": false}'

# CUA mode (manual server for development)
cd assets/templates/browserbase/cua && pnpm dev  # In separate terminal
prime eval run browser-cua-example -m openai/gpt-4o-mini -a '{"use_sandbox": false}'
  • All existing tests pass when running uv run pytest locally.
  • New tests have been added to cover the changes

Checklist

  • My code follows the style guidelines of this project as outlined in AGENTS.md
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • Any dependent changes have been merged and published

Additional Notes

CUA Server Deployment Options:

Mode Flag Startup Use Case
Pre-built image (default) None ~5-10s Production
Binary upload use_prebuilt_image=false ~30-60s Custom server
Manual server use_sandbox=false Instant Development

Future work:

  • Additional benchmark environments available on Prime Hub under browserbase/ org

---

## Bugbot Summary (updated)

```markdown


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Medium Risk**
> Introduces a new browser automation integration that spins up remote sandboxes, runs external HTTP servers, and adds new optional dependencies; failures could impact environment lifecycle/cleanup and test stability despite being largely additive.
> 
> **Overview**
> Adds a new `BrowserEnv` integration with two modes: **DOM mode** routes natural-language `navigate/observe/act/extract` calls through the Stagehand Python SDK, while **CUA mode** exposes coordinate-based primitives (`click`, `scroll`, `type_text`, etc.) backed by a CUA server.
> 
> CUA mode gains **automatic sandbox deployment** (including an optional prebuilt Docker image path vs binary upload vs local server), screenshot capture/retention controls, retry/health-check handling, and a backwards-compatible `CUASandboxMode` alias. The PR also vendors a TypeScript Fastify CUA server template (with Docker build/runtime + SEA binary scripts) and adds two example environments (`browser_dom_example`, `browser_cua_example`) plus docs, tests, and a new `[browser]` optional dependency group.
> 
> Additionally updates tool execution to allow tools to return structured multipart content (lists) instead of forcing `str(...)`, and adjusts import tests to tolerate missing optional dependency imports.
> 
> <sup>Written by [Cursor Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit 907fddd8bd0a847196c1fdbe9dc325eaf37a8e43. This will update automatically on new commits. Configure [here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

@CLAassistant
Copy link

CLAassistant commented Jan 15, 2026

CLA assistant check
All committers have signed the CLA.

@filip-michalsky filip-michalsky changed the title ruff precommit Add Browser Env Integration Jan 15, 2026
@filip-michalsky filip-michalsky marked this pull request as ready for review January 16, 2026 16:02
filip-michalsky and others added 8 commits January 16, 2026 16:31
* use remote sandbox env for cua mode

* update tests

* remote cua in sandbox

* Fm/browser add binary (#4)

* binary

* update

* fix non binary execution

* fix ruff

* update default flags
}

# Thread-safe locks (shared)
self._thread_lock = threading.Lock()
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Redundant threading lock in async-only context

Low Severity

_thread_lock (a threading.Lock) wraps an asyncio.Lock in _get_http_client, which is an async method that will only be called from async contexts. The threading lock is unnecessary since the async lock already provides sufficient protection for concurrent coroutines. Additionally, holding a synchronous lock while awaiting an async lock is an anti-pattern that can cause blocking issues.

Additional Locations (1)

Fix in Cursor Fix in Web

@cdreetz
Copy link
Collaborator

cdreetz commented Jan 29, 2026

@willccbb wanna take a look? everything for dom should be good to go

idk how to get ty check to pass tho? checked merged prs and they pass with warnings but this one fails with warnings

browserbaseProjectId?: string;
viewport?: Viewport;
proxies?: boolean;
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CUA mode silently ignores advanced_stealth parameter

Medium Severity

The advanced_stealth parameter is accepted by BrowserEnv and CUAMode in Python, which includes it in session_config as browserSettings.advancedStealth. However, the TypeScript CUA server's SessionCreateRequest type doesn't define browserSettings, and sessionManager.ts constructs browserSettings with only viewport, ignoring any advancedStealth setting. Users enabling advanced_stealth=True in CUA mode expect Browserbase's anti-bot detection mode, but it has no effect. DOM mode correctly handles this parameter.

Additional Locations (1)

Fix in Cursor Fix in Web

Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 2 potential issues.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.



def load_environment(
project_id: str,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DOM example requires project_id but documentation omits it

High Severity

The load_environment function requires project_id: str as a mandatory positional argument with no default value. However, the README shows usage without passing this argument (prime eval run browser-dom-example -m gpt-4.1-mini ...), and the docstring example shows env = load_environment() with no arguments. Running the example as documented will raise a TypeError. The CUA example correctly uses optional parameters with defaults, but the DOM example is inconsistent.

Additional Locations (1)

Fix in Cursor Fix in Web

"proxies": proxies,
"browserSettings": {"advancedStealth": advanced_stealth}
if advanced_stealth
else None,
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CUA mode advanced_stealth parameter is silently ignored

Medium Severity

When advanced_stealth=True is passed to BrowserEnv in CUA mode, the Python code sends browserSettings: {"advancedStealth": true} in the session config. However, the TypeScript CUA server's SessionCreateRequest interface doesn't include browserSettings, and sessionManager.ts constructs its own browserSettings object containing only viewport settings. The advancedStealth setting is silently discarded, so users expecting anti-bot detection will not have it enabled.

Additional Locations (1)

Fix in Cursor Fix in Web

@willccbb willccbb merged commit c1e1a5d into PrimeIntellect-ai:main Jan 30, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants