fix(server): Pi now sees screenshots returned by T3 MCP tools

[anntnzrb]

Problem

Pi drops image blocks from T3 MCP tool results. With preview_snapshot (and the device screenshot tool), the model receives the page metadata but never the screenshot pixels, even on an image-capable model. Fixes #15869.

The Pi bridge extension's formatMcpContent keeps only text blocks and structuredContent, and the registered tool's execute wraps that string as the only tool-result block (content: [{ type: "text", text }]). An image-only result is worse: it falls back to JSON.stringify(result), so the model gets the base64 as text.

Change

formatMcpContent becomes mcpToolContent, which returns Pi tool-result content directly:

• Text and structuredContent are joined into one text block, exactly as before.
• Well-formed MCP image blocks (data and mimeType are strings) follow as Pi image blocks, the shape Pi's AgentToolResult.content accepts.
• An image-only result returns just the image instead of its JSON.
• A text-only result is unchanged.

Pi already replaces tool-result images with a placeholder for models without image input (packages/ai/src/api/transform-messages.ts), so this needs no capability check on our side. T3's own UI is unaffected: PiAdapterV2's contentText reads only text blocks, so no image bytes go over the websocket.

The diff is limited to the extension source and its test. Pi is the only provider that reaches T3's MCP server through this T3-owned bridge; the others connect with their own MCP clients.

Scope and approval

This is the remaining gap from #13777, which ported Pi to V1 and also fixed this. It was closed after the V2 transition, and the closing comment asks for "a focused PR built on current main" for any gap that remains in the shipped implementation. The bug is tracked in #15869. That issue has not been triaged yet, so I'm submitting this under the small-fix exception for an obvious bug: preview_snapshot builds an image block for the model (McpHttpServer.ts), and the Pi bridge discards it. The fix converts one function's output and changes no contract, setting, or default.

Verification

Regression test. apps/server/src/orchestration-v2/Adapters/piT3McpExtensionSource.test.ts loads the shipped extension source with a stub MCP endpoint, then calls the registered tool's execute. The tests cover four cases: text plus image, text only, image only, and a malformed image block.

vp test run src/orchestration-v2/Adapters/piT3McpExtensionSource.test.ts

• On main source: Tests 2 failed | 5 passed (7). The failures are text plus image (expected 1 to equal 2 content blocks) and image only (the image came back as a text block).
• With this change: Tests 7 passed (7). piT3McpInjection.test.ts also passes (12/12 across both files).

End-to-end with real Pi. Real Pi 1.0.2 loaded the generated extension against a local stub MCP server whose preview_snapshot returns the same shape as McpHttpServer (text, structuredContent, and a 64×64 solid-red PNG). The model was claude-haiku-4-5, and the prompt asked for the screenshot's dominant color, or NO_IMAGE if the result had none:

Extension source	Model answer
main	NO_IMAGE
this PR	Red

I also ran tsc --noEmit for apps/server (clean) and vp lint / vp fmt --check on both changed files.

Not checked: a full T3 desktop/web turn calling the real preview_snapshot against a live preview browser. The e2e run above used Pi's real extension loading and tool-result path, with only the MCP server stubbed.

Made by Claude Opus 5.5 in omp.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

🧰 Additional context used

📚 Code guidelines (1)

docs/internals/effect-services.md — auto-discovered

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: Repository: pingdotgg/t3code/.coderabbit.yaml
• Review profile: CHILL
• Plan: Advanced
• Run ID: 34461dde-cd3e-41de-9b5e-5e630b400720

📥 Commits

Reviewing files that changed from the base of the PR and between a1d9d72 and af9ad9d.

📒 Files selected for processing (2)

• apps/server/src/orchestration-v2/Adapters/piT3McpExtensionSource.test.ts
• apps/server/src/orchestration-v2/Adapters/piT3McpExtensionSource.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The Pi MCP extension converts MCP tool results into Pi content blocks. It preserves valid image blocks alongside text and structured content. Tests cover text-only, image-only, and image blocks without a MIME type.

Changes

Pi MCP result content

Layer / File(s)	Summary
Map MCP results to Pi content apps/server/src/orchestration-v2/Adapters/piT3McpExtensionSource.ts, apps/server/src/orchestration-v2/Adapters/piT3McpExtensionSource.test.ts	Tool execution now uses mcpToolContent to return content blocks. The converter handles nullish values, primitives, text, structured content, and valid images. Tests check content ordering and shapes, and confirm that an image without a MIME type is omitted.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~15 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: juliusmarminge

Merge Risk: ⚪ Minimal · up to af9ad

The change can proceed through normal checks; no concrete merge-blocking issue remains.

Security Architecture Review

Security architecture risk: 🔵 Low · up to af9ad

Screenshot pixels now accompany existing tool results without changing access controls or tool permissions. Risk is low, but end-to-end image handling and responses to untrusted visual content are not fully validated.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The additional information exposure is image content returned by existing MCP calls within the provider session. Screenshot pixels can contain sensitive or attacker-controlled visual content, but the PR does not introduce a new endpoint, destination selector, or credential authority.

Trust Boundaries and Controls

• observed — The existing provider-scoped bearer authentication, current-session credential injection, MCP request headers, and Pi tool-call approval hook remain unchanged. Image preservation changes returned data rather than these authority controls.
• observed — The inspected tool-event and thread-snapshot projections extract text blocks through contentText. They do not forward the new image blocks into those T3 UI outputs; this does not establish provider-side storage or retention behavior.

Hardening Proposals

• proposed — Validate the real screenshot serialization path and a screenshot containing adversarial visual instructions, including any subsequent tool call, to confirm image compatibility and continued enforcement of existing approval boundaries.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: Pi now receives screenshots returned by T3 MCP tools.
Description check	✅ Passed	The description covers the Problem, Change, Scope and approval, and Verification sections. It explains the bug and fix, provides regression and end-to-end results, and states what was not checked.
Linked Issues check	✅ Passed	Issue #15869 requires valid MCP image blocks to reach Pi as image tool-result content while text-only results remain text-only. mcpToolContent preserves well-formed image blocks, returns image-only …
Out of Scope Changes check	✅ Passed	The source change and regression tests are limited to the Pi MCP result conversion required by #15869. No unrelated changes appear in the reviewed diff.

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}