Integrate Shelley coding agent as local sidecar#3
Conversation
- Enabled window decorations in the Tauri configuration for a more polished UI. - Refactored screenshot capture logic to streamline handling for vision-capable models. - Removed redundant image generation model checks from various components to simplify code. - Enhanced navigation component styles for better visual consistency.
- Revised sections to reflect the new vision capabilities, emphasizing automatic screenshot capture for all agent calls. - Changed terminology from "image generation models" to "vision support" for clarity. - Removed outdated references to model-specific screenshot behavior and added details on universal support and privacy measures. - Updated UI descriptions in the README to align with recent changes in functionality.
Ship Shelley (boldsoftware/shelley) inside the t2t app as a Tauri sidecar binary. Adds a "Chat" tab in the settings window that embeds Shelley's web UI via iframe, and routes fn+ctrl agent mode through Shelley when running (falling back to MCP/AppleScript agents if Shelley is unavailable). - Add Chat tab with MessageCircle icon to Nav and settings page - Configure Tauri externalBin for shelley sidecar binary - Add shell plugin for sidecar process management - Rust: spawn Shelley on app start, kill on exit, health check API - Pass ANTHROPIC_API_KEY/OPENROUTER_API_KEY to Shelley via env vars - SQLite database stored in app data dir (shelley.db) - fn+ctrl now tries Shelley first, opens Chat tab on success - Add send_to_shelley command for programmatic conversation creation https://claude.ai/code/session_019BM8jk4dqoSyrfzrwNAMj3
Visual feedback flow for Shelley agent mode: - fn held → green bar (typing mode, existing) - fn+ctrl held → purple bar (agent mode, existing) - Release → amber bar (transcribing, existing) - Shelley dispatched → orange bar with sweeping light animation - Orange bar persists until user clicks it → opens Chat tab - Escape also dismisses the Shelley indicator Key changes: - shelleyActive state: independent of processing, persists after HTTP POST returns since Shelley works asynchronously in background - Orange bar (#f97316) with animated sweep distinguishes from amber processing - Bar becomes clickable (pointer-events enabled) during agent processing - open_chat_window Tauri command: opens settings window, switches to Chat tab - fetch-shelley.sh: auto-download script for the Shelley binary - .gitignore: exclude large Shelley binary from repo https://claude.ai/code/session_019BM8jk4dqoSyrfzrwNAMj3
README: restructured as tutorial → how-to → reference → dev setup. Removed explanation bloat (vision technical details, history schema, screenshot encoding). Added Shelley, Chat tab, correct status bar colors, updated agent fallback chain, env vars table, file locations. docs/+page.svelte: fixed red→green bar, cyan→purple bar, added Shelley as primary agent with MCP/AppleScript as fallbacks, documented orange sweep animation and clickable bar behavior. https://claude.ai/code/session_019BM8jk4dqoSyrfzrwNAMj3
Wrap set_activation_policy calls in open_chat_window and tray menu handler with #[cfg(target_os = "macos")] so the build compiles on Linux. https://claude.ai/code/session_019BM8jk4dqoSyrfzrwNAMj3
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9668184096
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (result.running && result.port) { | ||
| shelleyUrl = `http://localhost:${result.port}`; |
There was a problem hiding this comment.
get_shelley_status can report running: true with an error when the port is not ready yet, but this branch treats any running && port as healthy and immediately loads http://localhost:<port>. In that startup window the iframe points at an unavailable endpoint, so users see a broken embedded page instead of the intended “starting up”/error state and retry flow.
Useful? React with 👍 / 👎.
|
|
||
| let conv_id = conv.get("conversation_id") | ||
| .or_else(|| conv.get("id")) | ||
| .and_then(|v| v.as_str().or_else(|| v.as_i64().map(|_| ""))) |
There was a problem hiding this comment.
The async Shelley command converts numeric IDs with v.as_i64().map(|_| ""), which drops the actual value and builds a chat URL like /conversation//chat whenever Shelley returns an integer id. That causes the follow-up chat request to fail for numeric-ID responses; the synchronous path already handles this correctly by converting numbers to strings.
Useful? React with 👍 / 👎.
| cmd.stdin(Stdio::null()) | ||
| .stdout(Stdio::null()) | ||
| .stderr(Stdio::piped()); |
There was a problem hiding this comment.
This spawns Shelley with stderr piped but never reads from the pipe, so if Shelley emits enough stderr logs, the pipe buffer can fill and block the child process. In that state the sidecar can stall and agent-mode requests will start timing out/failing under verbose logging conditions.
Useful? React with 👍 / 👎.
2 participants
Summary
This PR integrates Shelley, a local coding agent, as a bundled sidecar process into t2t. When users hold
fn+ctrlto enter agent mode, their voice commands are now routed to Shelley first, with fallback to MCP servers and AppleScript if Shelley is unavailable.Key Changes
fetch-shelley.shscript to download pre-built Shelley binaries or build from source viago installtauri-plugin-shellto enable spawning the Shelley sidecar processANTHROPIC_API_KEYandOPENROUTER_API_KEYvia environment variables; database stored in app data dirImplementation Details
start_shelley()and stopped on exitfind_available_port()to bind to an available TCP portPOST /conversationsandPOST /conversation/{id}/chat)send_to_shelley_sync()is used in the Fn key handler thread; asyncsend_to_shelley()is available for Tauri commandsFiles Modified
.gitignore: Ignore Shelley binariesREADME.md: Comprehensive rewrite with new structure and Shelley documentationdesktop/Cargo.toml: Addedtauri-plugin-shelldesktop/package.json: Added@tauri-apps/plugin-shelldesktop/capabilities/default.json: Added shell spawn permissionsdesktop/tauri.conf.json: Configured Shelley as external binarydesktop/src/main.rs: Shelley lifecycle, HTTP client integration, Tauri commandsdesktop/src/routes/+page.svelte: Orange bar, sweep animation, click-to-chat handlerdesktop/src/routes/settings/+page.svelte: Chat tab with Shelley iframedesktop/src/lib/components/Nav.svelte: Added "chat" tab typeweb/src/routes/docs/+page.svelte: Updated visual feedback and agent mode docsdesktop/scripts/fetch-shelley.sh: New script for binary managementdesktop/binaries/.gitkeep: New directory for bundled binarieshttps://claude.ai/code/session_019BM8jk4dqoSyrfzrwNAMj3