Refactor Tavily MCP configuration to shared workflow and add mcp-servers import support

[Copilot]

Overview

Refactored the Tavily MCP server configuration from the scout workflow into a separate shared workflow file and updated scout to use the imports field. Added full compiler support for importing mcp-servers entries from shared workflows. This change improves code reusability and maintainability by allowing MCP server configurations to be shared across multiple workflows.

Changes

1. Created Shared Tavily MCP Configuration

Created .github/workflows/shared/tavily-mcp.md containing the reusable Tavily MCP server configuration using mcp-servers:

---
mcp-servers:
  tavily:
    url: "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}"
    allowed: ["*"]
---

2. Updated Scout Workflow

Modified .github/workflows/scout.md to import the shared configuration instead of defining it inline:

---
name: Scout
# ... other config ...
engine: copilot
imports:
  - shared/tavily-mcp.md
tools:
  cache-memory:
    retention-days: 7
# mcp-servers section removed - now imported
---

3. Added Compiler Support for Importing MCP Servers

Schema Updates:

• Added full mcp-servers schema to included_file_schema.json with proper $defs for stdio_mcp_tool and http_mcp_tool (copied from main workflow schema)
• This ensures imported MCP servers are properly validated with the same strict schema as main workflows

Parser Updates (pkg/parser/frontmatter.go):

• Added MergedMCPServers field to ImportsResult struct
• Implemented extractMCPServersFromContent() to extract MCP servers from imported files
• Updated ProcessImportsFromFrontmatterWithManifest() to collect and merge MCP servers from imports

Compiler Updates (pkg/workflow/compiler.go):

• Implemented mergeMCPServers() to merge imported MCP servers with top-level ones using object maps
• Updated workflow compilation to merge imported MCP servers with main workflow MCP servers
• Handles newline-separated JSON objects from multiple imports

4. Fixed Critical Bug in Import Merging

Discovered and fixed a bug in pkg/workflow/compiler.go where the mergeTools() function failed to parse tools from multiple imports. The function was attempting to parse concatenated newline-separated JSON objects as a single JSON object, which is invalid:

{"tool1": {...}}
{"tool2": {...}}

The Fix: Modified mergeTools() to:

1. Split the input string by newlines
2. Parse each JSON object individually
3. Iteratively merge them using parser.MergeTools()

This fix enables workflows to properly import and merge both tools and mcp-servers from multiple shared configuration files.

5. Updated Documentation

Include Directives Documentation:

• Updated frontmatter merging section to mention both tools: and mcp-servers:
• Added example showing MCP server merging with Tavily

Packaging and Imports Guide:

• Updated frontmatter merging section with MCP server example
• Added new Example 3 showing MCP server imports via the imports: field
• Enhanced imports field documentation to mention MCP servers

6. Added Integration Tests

Added comprehensive tests in pkg/workflow/imports_test.go to prevent regressions:

• TestCompileWorkflowWithImports - validates single import functionality for tools
• TestCompileWorkflowWithMultipleImports - validates multiple tool imports merge correctly
• TestCompileWorkflowWithMCPServersImport - validates MCP server imports work correctly

7. Merged Main Branch

• Merged latest changes from main branch
• Recompiled all workflows to update lock files with latest changes
• All tests passing including import-related tests

8. Code Organization and Linting

• Created pkg/workflow/imports.go to consolidate import/merge related functions
• Moved mergeTools() and mergeMCPServers() from compiler.go to imports.go
• Fixed linting issue: removed unnecessary nil check (S1031)
• All linting checks passing

Verification

• ✅ Scout workflow compiles successfully with Tavily MCP server imported from shared file
• ✅ All existing tests pass
• ✅ New integration tests validate imports functionality for both tools and mcp-servers
• ✅ Lock file contains correct MCP configuration:

"tavily": {
  "type": "http",
  "url": "https://mcp.tavily.com/mcp/?tavilyApiKey=${{ secrets.TAVILY_API_KEY }}",
  "tools": ["*"]
}

• ✅ Successfully merged main branch with no conflicts
• ✅ All workflows recompiled successfully
• ✅ All linting checks passing
• ✅ Code properly organized with imports functionality in dedicated file

Impact

This change enables reusable MCP server configurations across multiple workflows using the mcp-servers field in shared workflow files, improving maintainability and consistency. It also fixes a fundamental issue with the imports system that would have affected any workflow using multiple imports.

Original prompt

Refactor Tavily MCP configuration in a separate shared workflow and import in scout using "imports" field in front matter.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[pelikhan]

@copilot the compiler should support importing mcp-servers entries. move tavily under mcp-servers and add support in compiler to import mcp servers entries as well

---

🤖 Workflow run triggered by this comment

[Copilot]

Done! I've moved tavily under mcp-servers in the shared file and added full compiler support for importing mcp-servers entries.

Changes in commit 7892cf8:

• Updated shared/tavily-mcp.md to use mcp-servers instead of tools
• Added mcp-servers to the included file schema
• Implemented extractMCPServersFromContent() and mergeMCPServers() functions
• Updated the parser and compiler to merge imported mcp-servers with top-level ones
• Added test case TestCompileWorkflowWithMCPServersImport to validate the functionality

The scout workflow now correctly imports the Tavily MCP server from the shared file using mcp-servers, and the compiled lock file contains the proper HTTP MCP server configuration.

[pelikhan]

@copilot update docs on imports to mention MCP servers

[Copilot]

@copilot update docs on imports to mention MCP servers

Done! Updated documentation to mention MCP servers can be imported:

include-directives.md:

• Updated frontmatter merging section to mention both tools: and mcp-servers:
• Added example showing Tavily MCP server merging

packaging-imports.md:

• Updated frontmatter merging with MCP server example
• Added Example 3 demonstrating MCP server imports via the imports: field
• Enhanced imports field documentation

Commit d413d04

[pelikhan]

@copilot merge main, recompile

---

🤖 Workflow run triggered by this comment

[Copilot]

@copilot merge main, recompile

---...

Done! Merged main branch and recompiled all workflows.

Changes:

• Merged latest commits from main (including new issue-classifier workflow and copilot log parsing improvements)
• Recompiled all 9 workflows successfully
• All import-related tests passing
• Tavily MCP server still correctly configured in scout.lock.yml

Commit 9f3a0b6

[pelikhan]

@copilot

• lint and fix linting issues
• move code related to import/merge of share workflows to "imports.go"

---

🤖 Workflow run triggered by this comment

[Copilot]

@copilot

• lint and fix linting issues
• move code related to import/merge of share workflows to "imports.go"...

Done!

Linting:

• Fixed linting issue S1031 (unnecessary nil check in mergeMCPServers)
• All golangci-lint checks passing

Code Organization:

• Created pkg/workflow/imports.go with import/merge related functions
• Moved mergeTools() and mergeMCPServers() from compiler.go to imports.go
• All tests passing

Commit 1f8e0a5