feat: Word orchestrator, Outlook skills, WorkIQ integration#45
Merged
feat: Word orchestrator, Outlook skills, WorkIQ integration#45
Conversation
- Add 'outlook' to OfficeHostApp and AgentHost type unions - Add Outlook detection in detectOfficeHost() - Create Outlook agent with read/compose mode guidance - Create Outlook app prompt - Create 9 Outlook tools (get_mail_item, get_mail_body, get_mail_attachments, set_mail_body, set_mail_subject, add_mail_recipient, reply_to_mail, forward_mail, get_user_profile) - Wire tool routing, system prompt, and agent service for outlook host - Create separate Outlook manifest (MailApp type) - Add outlook tool limit test Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…mEdit form) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
OfficeRuntime.storage requires SharedRuntime which Outlook doesn't support. Without this fallback, Zustand hydration hangs and the UI is stuck on 'Initializing...'. Now gracefully falls back to localStorage. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Add optional 'hosts' field to SkillMetadata - Parse 'hosts' in skill frontmatter - Filter getSkills() and buildSkillContext() by host - Filter SkillPicker bundled/imported lists by detected host - Add hosts: [excel] to Excel skill frontmatter Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…Word Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New tools: get_attachment_content, add_file_attachment, remove_attachment, get_mail_categories, set_mail_categories, remove_mail_categories, add_notification, remove_notification, save_draft, get_mail_headers, display_new_message, display_new_appointment, get_diagnostics Also updated Outlook skill with full tool guidance table. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Queries Exchange calendar via makeEwsRequestAsync with SOAP FindItem. Returns subject, time, location, organizer for appointments in a date range. Defaults to today if no dates specified. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Legacy Exchange tokens are disabled in most M365 tenants since 2025. Added clear error message explaining the limitation and suggesting Microsoft Graph API with nested app authentication as alternative. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Created src/skills/powerpoint/SKILL.md with tool guidance and workflows - Registered PowerPoint skill in skillService (host-filtered) - Expanded POWERPOINT_APP_PROMPT.md with detailed tool selection guide, PptxGenJS reference, and constraints documentation - Removed get_appointments EWS tool from Outlook (legacy tokens deprecated) - Updated Outlook skill to remove calendar tool reference Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add nested VersionOverridesV1_1 with SupportsPinning=true for both MessageRead and MessageCompose extension points. This keeps the task pane open when switching between emails in Outlook. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Recursively traverse group shapes to extract nested text content - Detect SmartArt/graphics and hint to use get_slide_image as fallback - Update agent instructions to always try image capture when text is empty Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Reverted to simpler text extraction approach without loading shape type (which fails for certain shape types like SmartArt). Group shape text extraction now uses try-catch per shape instead of type checking. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The batch context.sync() fails when any shape has an unsupported textFrame. Now each slide is processed in its own try-catch block so one problematic slide doesn't break the entire presentation read. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Process each shape with individual context.sync() + try-catch so
one unsupported textFrame doesn't poison the batch
- Pass options object {width} to getImageAsBase64 instead of raw number
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Base64 PNG at 800px was 72KB+ which exceeds Copilot CLI tool output limit. Reduced default to 320px, capped max at 400px, and added size warning when output exceeds 50KB. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Agent now explicitly follows a create→verify→evaluate→refine loop for every slide modification. Expects 2-3 iterations per slide instead of one-shot output. Updated both AGENT.md and SKILL.md. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New tools: get_slide_shapes, get_slide_layouts, get_selected_slides, get_selected_shapes, add_geometric_shape, add_line, delete_slide, move_slide, set_slide_background, apply_slide_layout, set_shape_text, update_shape_style, move_resize_shape, delete_shape. Now supports: shape styling (fill/outline/font), shape positioning, slide management (delete/reorder/background/layouts), geometric shapes, lines, selection queries, and granular shape editing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Updated agent instructions and skill to ALWAYS call get_selected_slides as the very first step so the agent knows which slide the user is on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
slide.index is 1-based (what the user sees in PowerPoint) but all tools expect 0-based slideIndex. Now returns both so the agent uses the correct index and communicates the correct number to the user. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New tools (10→20 total): insert_paragraph, insert_break, apply_paragraph_style, set_paragraph_format, get_document_properties, insert_image, get_comments, insert_list, get_content_controls, insert_text_at_bookmark. Created Word skill (SKILL.md) with tool guidance and workflows. Updated agent with iterative refinement loop and selection awareness. Expanded app prompt with tool selection guide and HTML formatting tips. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
New tools: get_headers_footers, set_header_footer, get_table_data, add_table_rows, add_table_columns, delete_table_row, set_table_cell_value, insert_hyperlink, insert_footnote, insert_endnote, get_footnotes_endnotes, delete_content, insert_content_control, format_found_text, get_sections. Updated Word skill with new tool guidance and workflows. Added word/powerpoint host tool limit tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…, and formatting rules Adapted from reference presentation editing skill patterns: - Layout variety mandate (never >2 consecutive same layouts) - Iterative verification loop (create → verify → fix → re-verify) - Template adaptation pitfalls (content count/length mismatch) - PptxGenJS formatting rules (bold, bullets, colors, margins, fonts) - Verification checklist (overlap, overflow, spacing, contrast) Added tests-pptx/test-agent.mjs for mock-based agent quality testing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Uses spire.presentation to render slides to PNG, then PIL to create a labeled grid image. No LibreOffice dependency. Usage: python3 tests-pptx/thumbnails.py tests-pptx/test.pptx Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…slides After visual QA of generated slides, the main defect was text overflow (content cut off at box edges). Added: - Font size guidelines per element type (title, body, cards, tables) - Content limits per slide type (max bullets, max columns) - Space budget calculations (title zone, content zone) - Mandatory get_slide_image verification after EVERY slide creation - Bottom buffer rule (0.3" margin) Reduced default font sizes in examples: bullets 18→16pt, tables 14→13pt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…int slides Visual QA showed bold labels merging into descriptions without spacing (e.g., 'Machine LearningSystems that...'). Added explicit rules and examples to all three PowerPoint prompt files (APP_PROMPT, AGENT, SKILL): - Never merge bold label + description in one text run - Use colon separator in same line OR indented sub-line - Added WRONG/CORRECT code examples showing the issue and fix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
… PowerPoint
Visual QA round 2 showed slides 2 & 4 rendering '[object Object],[object Object]'
instead of text. Root cause: the nested text array syntax { text: [{...}, {...}] }
is not supported by PptxGenJS's addText API.
Replaced all examples with two safe patterns:
- Option A: Single string with colon separator ('Label: description')
- Option B: Bold heading + indented sub-line (indentLevel: 1)
Added explicit WRONG example showing the nested array anti-pattern.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Split the monolithic PowerPoint skill (239 lines) into focused skills: - powerpoint (core): Tool routing, operating loop, always-on defaults - powerpoint-deck-builder: Layout variety, content sizing, verification loops - powerpoint-redesign: Template adaptation, shape manipulation, redesign workflows - powerpoint-formatting: PptxGenJS rules, anti-patterns, correct code examples Users can now toggle skills per task — deck creation gets sizing rules while simple reads don't get bloated context. All skills are host-filtered to powerpoint. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
QA round 3 still showed bold labels merging into descriptions ('Machine
LearningSystems that...'). Root cause: the agent uses separate text runs
(bold + normal) which PptxGenJS renders without spacing.
Removed the indented sub-line option (Option B) from all prompts.
Only ONE pattern is now allowed: 'Label: Description' as a single string.
Added explicit anti-pattern example showing the separate runs bug.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…cription QA round 4 showed Slide 2 overflow: agent wrote full sentences as bullet descriptions instead of short phrases. Tightened rules across all prompts: - Bullets: max 8 words (was 10), max 5 bullets (was 5-6) - Label:Description pattern: description ≤ 6 words after colon - New 'definition + bullets' combo limit: 2-line paragraph + 4 bullets - Marked content limits as MANDATORY with warning - 'Shorten the text' priority over shrinking fonts Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Major prompt overhaul across all PowerPoint skills: Verification loop: - Changed from 'recommended' to MANDATORY with checklist format - Added per-slide loop algorithm: create → verify → fix → re-verify - Added fix action table (issue → specific fix) - Added minimum get_slide_image call counts (= number of slides) - 'Do NOT move to next slide until current passes all checks' Content limits: - Definition + bullets: max 4 bullets (NOT 5) when intro exists - German-language examples in ✅/❌ patterns - Word-breaking fix actions: reduce columns (4→3) or shorten text Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
QA round 8: German compound words ('Medikamentenentwicklung', 22 chars)
break mid-word in 3-column layouts. Added:
- No word in a column should exceed 12 characters
- Explicit compound word → shorter synonym examples in fix table
- 'Visual clarity > terminological precision' principle
- Bottom text cut-off fix action
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removed excessive content limit rules (word counts, char limits, space budgets). Kept the essential formatting rules (PptxGenJS anti-patterns) and a compact font size table. The core improvement: a much clearer Create → Verify → Fix loop that emphasizes 'look at it and fix what you see' over memorizing rules. APP_PROMPT: ~95 lines (was ~119) AGENT.md: ~60 lines (was ~85) deck-builder SKILL: ~70 lines (was ~136) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The agent does verification loops now but misses text cut off at the bottom of text boxes. Added 'CHECK THE BOTTOM EDGE FIRST' as step 1 in the verification checklist across all prompts — this is where overflow always happens and is easy to overlook. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The agent couldn't detect text overflow in small cards because the image was only 320px wide (max 400px). At that resolution, cut-off text at the bottom of cards is nearly invisible. Changes: - Default width: 320 → 800px - Max width: 400 → 1200px - Size warning threshold: 50KB → 200KB - Updated description to emphasize overflow detection purpose Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…tection Instead of sending one large full-slide image, the agent can now request specific regions: full, top, bottom, left, right. The 'bottom' region captures the bottom half at 960px (high detail) and crops via Canvas, keeping image size small while giving the agent a zoomed view of where text overflow typically occurs. - Added region parameter (enum: full/top/bottom/left/right) - cropImage() uses Canvas to crop the captured PNG - 'full' captures at 600px, regions at 960px (cropped = ~half the data) - Auto-fallback to 400px if result still exceeds 150KB - Updated all prompts to use get_slide_image(region: 'bottom') in the verification loop Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…4:3 dimensions The agent was generating slides at 10"×7.5" (4:3 default) regardless of the actual presentation format. On 16:9 presentations (13.33"×7.5"), content only filled 75% of the slide width. Changes: - add_slide_from_code now reads pageSetup.slideWidth/Height and sets PptxGenJS layout to match (falls back to 10×7.5 on older API) - get_presentation_overview now reports slide dimensions + aspect ratio - Updated all prompts/skills to use dynamic widths instead of hardcoded w:9 / right≤9.5 (all examples now show 16:9 values with note to check get_presentation_overview for actual size) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…zoom One tool call now returns 5 images: a 480px overview + 4 zoomed quadrant crops from a 960px capture (top-left, top-right, bottom-left, bottom-right). Each quadrant is 480×360px — high enough detail to catch text overflow while keeping individual images small. "detailed" is the new default. The agent inspects all 5 images in the verification loop to catch issues anywhere on the slide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Returning 5 base64 images in one tool result would exceed the Copilot CLI output size limit. Instead, the verification loop now uses separate calls: full overview + bottom-left/bottom-right quadrants for 2x zoom where overflow occurs. Each call returns one image, staying well within size limits. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Updated intro, diagram, features, prerequisites to reflect all 4 hosts - Added tool counts: Excel ~83, PowerPoint 24, Word 35, Outlook 22 - Updated test counts to current (31 files, 276 unit tests) - Noted E2E tests for non-Excel hosts are planned Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…y loop Raised recommended font sizes (body 16-20pt, cards 14-16pt, tables 13-15pt, min 13pt). Added 'text too small' as explicit check in the Create→Verify→Fix loop so the agent catches and fixes tiny text. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The user only sees tool calls in the UI. The agent now narrates each step: which slide it's working on, what layout it chose, what it found during verification, and what it's fixing. Examples in the user's language keep it natural. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Removed turnAnchor='top' from ThreadPrimitive.Viewport which was pinning the scroll position to the top of each turn. Now the viewport follows new content as it streams in, so the user can see progress narration and tool results in real time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a multi-session architecture for PowerPoint deck creation: - Planner session creates a structured JSON slide plan via submit_plan tool - Worker sessions execute slides sequentially, each with a fresh context - Automatic retry on worker failure (1 retry, then skip) - Progress narration streams to chat UI (plan/slide status/errors) New files: - src/lib/session-factory.ts — runSubSession() helper - src/tools/planner/index.ts — submit_plan tool + plan extraction - src/services/ai/prompts/PLANNER_PROMPT.md — planning-only prompt - src/services/ai/prompts/WORKER_PROMPT.md — single-slide worker prompt - src/hooks/useDeckOrchestrator.ts — orchestrateDeck() function Modified: - src/hooks/useOfficeChat.ts — detects multi-slide requests, routes to orchestrator instead of single session. Exposes clientRef. The orchestrator activates when: host=powerpoint AND prompt mentions N slides/folien AND doesn't say 'this slide'. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The submit_plan tool arguments weren't available in session events. Now the tool handler captures the plan directly via a module-level variable, and the orchestrator reads it via getLastPlan(). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fast mode (default): 3 slides per worker session — fewer sessions, faster Deep mode: 1 slide per worker — maximum quality, full verify per slide Triggered by keywords in prompt: 'deep', 'detail', 'qualit' → deep mode. Otherwise defaults to fast mode. 10 slides: fast = 4 sessions (1 planner + 3 workers), deep = 11 sessions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Fast = 1 planner + 1 worker session (all slides). Deep = 1 planner + N worker sessions (1 slide each). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: add WorkIQ toggle and MCP tool integration Add a one-click WorkIQ toggle in the ChatHeader that enables Microsoft 365 data access (emails, meetings, documents, Teams) via the WorkIQ MCP server. Also adds full MCP tool integration supporting HTTP, SSE, and stdio transports. Changes: - Add WorkIQ toggle button (brain icon) to ChatHeader with visual state - Add workiqEnabled setting to UserSettings with persistence - Add built-in WORKIQ_MCP_SERVER constant (no manual mcp.json import needed) - Add mcpClient.mjs server-side module for connecting to MCP servers - Add stdio transport support (StdioClientTransport) for subprocess-based MCP - Wire MCP servers into copilotProxy.mjs session.create with lifecycle cleanup - Pass active MCP servers from browser to proxy via websocket-client - Auto-inject WorkIQ when enabled in useOfficeChat hook - Wire McpManagerDialog into ChatHeader for advanced MCP server management - Update MCP config parser to accept stdio entries (command + args) - Update tests for stdio support (276 tests passing) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: deduplicate WorkIQ when both toggle and manual import exist Filter out any imported 'workiq' MCP server when the built-in toggle is active to prevent duplicate server connections. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: add stdio command allowlist, fix race condition, add WorkIQ tests - Add ALLOWED_STDIO_COMMANDS allowlist (npx, node, python, python3) to mcpClient.mjs — blocks arbitrary command execution via WebSocket - Fix __pending__ race condition in copilotProxy.mjs by using unique temp keys instead of a static '__pending__' key - Remove personal path (C:\Users\urstojki\...) from workiq-mcp.json - Add 8 unit tests: toggleWorkiq (4), WORKIQ_MCP_SERVER constant (4) - All 284 tests passing --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Torsten Mahr <torsten.mahr@gmail.com>
…#3) - Add Word document orchestrator (planner→worker pattern, mirrors PowerPoint) - Deep-mode triggers: deep keywords, multi-section requests, doc creation - WORD_PLANNER_PROMPT.md + WORD_WORKER_PROMPT.md with verify checklist - submit_document_plan tool + wordPlanner.ts - Add 3 Outlook skills: email-analysis, drafting, calendar - Add 3 Word skills: formatting, document-builder, tables - Add WorkIQ model dropdown in ChatHeader (separate model for WorkIQ sessions) - Add sideload scripts: start:excel, start:word, start:powerpoint - Improve prompt starters for all hosts - Add *.md?raw type declaration for Vite raw imports - Add docs/WORKIQ.md documenting WorkIQ integration and prerequisites - Fix wordPlanner test: pass mock invocation to handler - 47 new tests (wordDeepMode + wordPlanner) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…m PR #33 - Word: 25 new tools (paragraphs, tables, images, headers/footers, footnotes, content controls, hyperlinks, sections, bookmarks, formatting search) - Outlook: 22 tools for email, calendar, contacts, drafts, search - WorkIQ: Microsoft 365 data integration via MCP (toggleable, own model selector) - Word orchestrator: planner-worker pattern (wordPlanner, useDocumentOrchestrator) - PowerPoint deck orchestrator: useDeckOrchestrator hook - New skills: outlook, word, powerpoint skill files with host targeting - New agent: Outlook agent with frontmatter - New prompts: WORD_PLANNER_PROMPT, WORD_WORKER_PROMPT, WORKER_PROMPT, PLANNER_PROMPT - Updated ChatHeader: WorkIQ toggle + model selector + MCP button - Updated settings: workiqEnabled, workiqModel persisted in Zustand store - Updated useOfficeChat: WorkIQ MCP injection into active servers Co-authored-by: Torsten <trsdn@users.noreply.github.com>
…illPicker.tsx TS errors
…les and scripts
Merged
sbroenne
added a commit
that referenced
this pull request
Feb 26, 2026
Bumps version to **0.3.0**. ## What's new since v0.2.0 ### ✨ New Features - **Outlook host** — 22 tools (emails, calendar, contacts, folders, attachments, search, flags, drafts), dedicated agent, 4 skills, manifest (#45) - **Word document orchestrator** — planner→worker pattern with deep/fast modes, \submit_document_plan\ tool, Word expanded to 35 tools (#45) - **PowerPoint expanded** — 24 tools, 4 new skills (deck-builder, formatting, redesign, presentation), improved \get_slide_image\ with region cropping (#45) - **WorkIQ integration** — MCP stdio transport, workiqModel setting, UI toggle in ChatHeader (#45) - **MCP servers** — stdio/npx MCP server support + npm skill packages as internal tools (#41) - **Electron tray app** — system tray with single-instance lock, production server, installer packaging (#43) - **Session history** — persistent conversation history with restore picker (#43) - **Permissions flow** — interactive approve/deny permission UI (#43) - **Auto-scroll** — chat thread stays pinned to newest content (#43) - **Thinking indicator** — dynamic text from \ eport_intent\ and tool execution phase (#34, #37) - **Custom agents & skills** — ZIP import/export, management dialogs (#29, #38) - **Host-scoped skills** — skills filtered by current Office host (#40) ### 🔒 Security - bump rollup 4.58.0 → 4.59.0 (Arbitrary File Write CVE) (#46) - bump hono 4.12.0 → 4.12.3 (Auth Bypass CVE) (#46) ### 🛠 Other - Added \start:desktop:excel/ppt/word/outlook\ convenience scripts - Branch protection + squash-merge-only policy enforced (#44) - 458/458 integration tests passing; E2E: Excel ~187, PPT ~13, Word ~12, Outlook ~9 Co-authored-by: Stefan Broenner <stefan.broenner@microsoft.comm>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Takes ownership of and rebases PR #33 from @trsdn (Torsten) and @urosstojkic onto current main.
Originally submitted as sbroenne/office-coding-agent#33.
Word tools expanded to 35 (was 10), Outlook tools (22 new), WorkIQ integration,
Word/PPT orchestrators, new skills.
458/458 integration tests passing.
Co-authored-by: Torsten trsdn@users.noreply.github.com
Co-authored-by: urosstojkic urosstojkic@users.noreply.github.com