feat: Add StagehandCrawler with AI-powered browser automation (#1854)

### Description

Adds `StagehandCrawler` - a new browser crawler powered by
[Stagehand](https://www.browserbase.com/stagehand) that lets users
interact with pages using natural language instead of CSS selectors or
XPath. Extends `PlaywrightCrawler` and inherits all of its features:
routing, sessions, autoscaling, proxies, and navigation hooks.

- `StagehandPage` extends Playwright `Page` with four AI methods:
`act()`, `extract()`, `observe()`, and `execute()`.
- `StagehandOptions` configures the AI model, execution environment
(`LOCAL` / `BROWSERBASE`), and session parameters.
- `StagehandBrowserPlugin` and `StagehandBrowserController` integrate
Stagehand into the browser pool, managing session lifecycle and
fingerprint header injection.
- Because Stagehand controls the browser launch internally and
Playwright connects via CDP, only Chromium is supported, and browser
configuration is limited to the subset accepted by Stagehand's
`BrowserLaunchOptions`.
- Added a new guide covering basic usage, AI page operations, and
Browserbase integration.

### Issues

- Closes: #1738

### Testing

- Added unit tests for the `StagehandBrowserController`,
`StagehandBrowserPlugin`, and `StagehandCrawler` with Stagehand mocked
out - no real LLM connection required to run the test suite.

Author: Mantisus

docs/guides/architecture_overview.mdx

@@ -53,6 +53,8 @@ class PlaywrightCrawler
 
 class AdaptivePlaywrightCrawler
 
+class StagehandCrawler
+
 %% ========================
 %% Inheritance arrows
 %% ========================
@@ -63,6 +65,7 @@ BasicCrawler --|> AdaptivePlaywrightCrawler
 AbstractHttpCrawler --|> HttpCrawler
 AbstractHttpCrawler --|> ParselCrawler
 AbstractHttpCrawler --|> BeautifulSoupCrawler
+PlaywrightCrawler --|> StagehandCrawler
 ```
 
 ### HTTP crawlers
@@ -79,7 +82,10 @@ You can learn more about HTTP crawlers in the [HTTP crawlers guide](./http-crawl
 
 ### Browser crawlers
 
-Browser crawlers use a real browser to render pages, enabling scraping of sites that require JavaScript. They manage browser instances, pages, and context lifecycles. Currently, the only browser crawler is <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, which utilizes the [Playwright](https://playwright.dev/) library. Playwright provides a high-level API for controlling and navigating browsers. You can learn more about <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink>, its features, and how it internally manages browser instances in the [Playwright crawler guide](./playwright-crawler).
+Browser crawlers use a real browser to render pages, enabling scraping of sites that require JavaScript. They manage browser instances, pages, and context lifecycles. Crawlee provides two browser crawlers:
+
+- <ApiLink to="class/PlaywrightCrawler">`PlaywrightCrawler`</ApiLink> utilizes the [Playwright](https://playwright.dev/) library and provides a high-level API for controlling and navigating browsers. You can learn more about it in the [Playwright crawler guide](./playwright-crawler).
+- <ApiLink to="class/StagehandCrawler">`StagehandCrawler`</ApiLink> extends `PlaywrightCrawler` with AI-powered browser automation via [Stagehand](https://github.com/browserbase/stagehand). It adds natural-language methods (`act`, `extract`, `observe`, `execute`) directly on the page object. You can learn more about it in the [Stagehand crawler guide](./stagehand-crawler).
 
 ### Adaptive crawler
 
@@ -122,6 +128,12 @@ class AdaptivePlaywrightPreNavCrawlingContext
 
 class AdaptivePlaywrightCrawlingContext
 
+class StagehandPreNavCrawlingContext
+
+class StagehandPostNavCrawlingContext
+
+class StagehandCrawlingContext
+
 %% ========================
 %% Inheritance arrows
 %% ========================
@@ -143,6 +155,12 @@ PlaywrightPreNavCrawlingContext --|> PlaywrightCrawlingContext
 BasicCrawlingContext --|> AdaptivePlaywrightPreNavCrawlingContext
 
 ParsedHttpCrawlingContext --|> AdaptivePlaywrightCrawlingContext
+
+PlaywrightPreNavCrawlingContext --|> StagehandPreNavCrawlingContext
+
+StagehandPreNavCrawlingContext --|> StagehandPostNavCrawlingContext
+
+StagehandPostNavCrawlingContext --|> StagehandCrawlingContext
 ```
 
 They have a similar inheritance structure as the crawlers, with the base class being <ApiLink to="class/BasicCrawlingContext">`BasicCrawlingContext`</ApiLink>. The specific crawling contexts are:
@@ -154,6 +172,9 @@ They have a similar inheritance structure as the crawlers, with the base class b
 - <ApiLink to="class/PlaywrightCrawlingContext">`PlaywrightCrawlingContext`</ApiLink> for Playwright crawlers.
 - <ApiLink to="class/AdaptivePlaywrightPreNavCrawlingContext">`AdaptivePlaywrightPreNavCrawlingContext`</ApiLink> for Adaptive Playwright crawlers before the page is navigated.
 - <ApiLink to="class/AdaptivePlaywrightCrawlingContext">`AdaptivePlaywrightCrawlingContext`</ApiLink> for Adaptive Playwright crawlers.
+- <ApiLink to="class/StagehandPreNavCrawlingContext">`StagehandPreNavCrawlingContext`</ApiLink> for Stagehand crawlers before the page is navigated.
+- <ApiLink to="class/StagehandPostNavCrawlingContext">`StagehandPostNavCrawlingContext`</ApiLink> for Stagehand crawlers after the page is navigated.
+- <ApiLink to="class/StagehandCrawlingContext">`StagehandCrawlingContext`</ApiLink> for Stagehand crawlers.
 
 ## Storages
 

docs/guides/code_examples/playwright_crawler_stagehand/__init__.py

@@ -0,0 +1,47 @@
+import asyncio
+from typing import cast
+
+from crawlee.browsers import StagehandOptions
+from crawlee.crawlers import StagehandCrawler, StagehandCrawlingContext
+
+
+async def main() -> None:
+    crawler = StagehandCrawler(
+        stagehand_options=StagehandOptions(
+            model_api_key='your-openai-api-key',
+            model='openai/gpt-5.4-nano',
+        ),
+        max_requests_per_crawl=5,
+    )
+
+    @crawler.router.default_handler
+    async def handler(context: StagehandCrawlingContext) -> None:
+        context.log.info(f'Processing {context.request.url} ...')
+
+        # Dismiss overlays or interact with the page using natural language.
+        await context.page.act(input='Click the accept cookies button if present')
+
+        # Extract data from the page using AI.
+        extracted = await context.page.extract(
+            instruction='Get the page title and the main heading text',
+            schema={
+                'type': 'object',
+                'properties': {
+                    'title': {'type': 'string'},
+                    'heading': {'type': 'string'},
+                },
+            },
+        )
+
+        extract_result = extracted.data.result
+
+        if isinstance(extract_result, dict):
+            # Push extracted data to the dataset
+            # Use `cast()` to provide a more specific type hint for the extracted data.
+            await context.push_data(cast('dict[str, str | None]', extract_result))
+
+    await crawler.run(['https://example.com'])
+
+
+if __name__ == '__main__':
+    asyncio.run(main())

docs/guides/code_examples/playwright_crawler_stagehand/browser_classes.py

@@ -0,0 +1,37 @@
+import asyncio
+from typing import cast
+
+from crawlee.browsers import StagehandOptions
+from crawlee.crawlers import StagehandCrawler, StagehandCrawlingContext
+
+
+async def main() -> None:
+    # Use Browserbase cloud browser instead of a local Chromium instance.
+    crawler = StagehandCrawler(
+        stagehand_options=StagehandOptions(
+            env='BROWSERBASE',
+            browserbase_api_key='your-browserbase-api-key',
+            project_id='your-project-id',
+            model_api_key='your-openai-api-key',
+            model='openai/gpt-5.4-nano',
+        ),
+        max_requests_per_crawl=5,
+    )
+
+    @crawler.router.default_handler
+    async def handler(context: StagehandCrawlingContext) -> None:
+        context.log.info(f'Processing {context.request.url} ...')
+
+        extracted = await context.page.extract(
+            instruction='Get the main content of the page',
+        )
+
+        extract_result = extracted.data.result
+
+        await context.push_data(cast('dict[str, str | None]', extract_result))
+
+    await crawler.run(['https://example.com'])
+
+
+if __name__ == '__main__':
+    asyncio.run(main())