feat(mcp): Add searchable tool catalog

docs/adding-tools.md

@@ -6,12 +6,15 @@ Step-by-step guide for adding new tools to the Sentry MCP server.
 
 Not every tool is exposed to every consumer. We rely on several mechanisms to keep the active tool set manageable:
 
+- **Catalog by default** — Most tools are searchable/executable through `search_tools` + `execute_tool` automatically when experimental mode is enabled. Search uses the tool's existing name and description.
+- **Catalog registry** — `packages/mcp-core/src/tools/catalog/index.ts` lists ordinary Sentry operation tools. The catalog directory is intentionally flat: one tool entry per file.
+- **Special tools** — Wrapper/gateway tools (`search_tools`, `execute_tool`, `use_sentry`) live in `packages/mcp-core/src/tools/special/`. They still use the same tool types, but they are kept out of the ordinary catalog.
+- **Central direct exposure policy** — `packages/mcp-core/src/tools/surfaces.ts` lists the catalog tools that are also exposed directly through MCP `tools/list`.
 - **`requiredCapabilities`** — Tools declare which project capabilities they need (e.g. `profiles`, `replays`, `traces`). If the upstream project doesn't have a capability enabled, the tool is automatically hidden.
-- **`internalOnly`** — Composition primitives (e.g. `get_issue_details`, `get_trace_details`) that are only called by other tools like `get_sentry_resource`, never exposed directly via MCP.
 - **`experimental` / `hideInExperimentalMode`** — Feature flags for tools that are being tested or replaced.
 - **Skills & constraints** — The server filters tools based on granted skills and org/project constraints.
 
-We also expect upstream consumers (Claude Code plugins, Cursor, etc.) to use **tool selection** or **progressive disclosure** on their end. The total registered tool count can exceed what any single session needs because consumers pick a relevant subset.
+We also expect upstream consumers (Claude Code plugins, Cursor, etc.) to use **tool selection** or **progressive disclosure** on their end. The catalog can contain more tools than the direct MCP surface, but the registered top-level tool count must still stay within the limits below.
 
 ## Tool Count Limits
 
@@ -20,25 +23,40 @@ Target ~20 publicly visible tools. Never exceed 25. AI agents have limited tool
 Before adding a new tool, consider if it could be:
 1. Combined with an existing tool
 2. Implemented as a parameter variant
-3. Gated behind `requiredCapabilities` if only relevant to some projects
+3. Added to the searchable catalog instead of the top-level MCP surface
+4. Gated behind `requiredCapabilities` if only relevant to some projects
+
+### Choosing Direct Exposure
+
+After creating a tool module, add it to `packages/mcp-core/src/tools/catalog/index.ts`. Then update `packages/mcp-core/src/tools/surfaces.ts` only when it should be directly exposed through MCP `tools/list`:
+
+- Add high-frequency, foundational tools to `TOP_LEVEL_TOOL_NAMES`.
+- Leave long-tail tools out of `TOP_LEVEL_TOOL_NAMES` to make them available only through `search_tools` and `execute_tool` after the normal skill, constraint, experimental, and capability filters pass. The catalog gateway tools themselves are experimental for now.
+- Keep private implementation helpers as plain modules/functions instead of MCP tools.
+
+Do not add search-only summaries or catalog-only schemas. `search_tools` indexes the existing tool name and description.
 
 ## Tool Structure
 
 Each tool consists of:
-1. **Tool Module** - Single file in `src/tools/your-tool-name.ts` with definition and handler
-2. **Tests** - Unit tests in `src/tools/your-tool-name.test.ts`
+1. **Tool Module** - Single file in `src/tools/catalog/your-tool-name.ts` with definition and handler
+2. **Tests** - Unit tests in `src/tools/catalog/your-tool-name.test.ts`, including a baseline happy-path inline snapshot
 3. **Mocks** - API responses in `mcp-server-mocks`
 4. **Evals** - Integration tests (use sparingly)
 
+If a tool needs substantial helper code, put that code under
+`packages/mcp-core/src/tools/support/` and import it from the flat catalog tool
+file. Do not create per-tool subdirectories under `tools/catalog/`.
+
 ## Step 1: Create the Tool Module
 
-Create `packages/mcp-server/src/tools/your-tool-name.ts`:
+Create `packages/mcp-core/src/tools/catalog/your-tool-name.ts`:
 
 ```typescript
 import { z } from "zod";
-import { defineTool } from "./utils/defineTool";
-import { apiServiceFromContext } from "./utils/api-utils";
-import type { ServerContext } from "../types";
+import { defineTool } from "../../internal/tool-helpers/define";
+import { apiServiceFromContext } from "../../internal/tool-helpers/api";
+import type { ServerContext } from "../../types";
 
 export default defineTool({
   name: "your_tool_name",
@@ -168,7 +186,7 @@ See [common-patterns.md](common-patterns.md#response-formatting) for:
 
 Follow comprehensive testing patterns from `testing.md` for unit, integration, and evaluation tests.
 
-Create `packages/mcp-server/src/tools/your-tool-name.test.ts`:
+Create `packages/mcp-core/src/tools/catalog/your-tool-name.test.ts`:
 
 ```typescript
 import { describe, it, expect } from "vitest";
@@ -265,8 +283,10 @@ pnpm eval your-tool
 
 ## Checklist
 
-- [ ] Definition in `toolDefinitions.ts`
-- [ ] Handler in `tools.ts`
+- [ ] Tool module in `packages/mcp-core/src/tools/catalog/`
+- [ ] Tool registered in `packages/mcp-core/src/tools/catalog/index.ts`
+- [ ] Co-located test includes a baseline inline snapshot for the tool output
+- [ ] Direct exposure policy updated in `packages/mcp-core/src/tools/surfaces.ts` if this should be top-level
 - [ ] Unit tests with snapshots
 - [ ] Mock responses
 - [ ] 1-2 eval tests (if critical)
@@ -399,7 +419,7 @@ try {
 3. **Use structured outputs** - Define clear schemas for agent responses
 4. **Provide tool discovery** - Let agents explore available fields dynamically
 
-See `packages/mcp-server/src/tools/search-events/` and `packages/mcp-server/src/tools/search-issues/` for examples.
+See `packages/mcp-core/src/tools/catalog/search-events.ts` and `packages/mcp-core/src/tools/catalog/search-issues.ts` for examples. Their helper code lives under `packages/mcp-core/src/tools/support/`.
 
 ## Worker-Specific Tools
 
@@ -427,7 +447,7 @@ export default new Hono().post("/", async (c) => {
   return c.json({ results });
 });
 
-// In the MCP tool (tools.ts)
+// In the MCP tool module
 search_docs: async (context, params) => {
   const response = await fetch(`${context.host}/api/search`, {
     method: "POST",
@@ -454,6 +474,6 @@ This pattern works with both Cloudflare-hosted and stdio transports.
 
 ## References
 
-- Tool examples: `packages/mcp-server/src/tools.ts`
-- Schema patterns: `packages/mcp-server/src/schema.ts`
+- Tool examples: `packages/mcp-core/src/tools/catalog/`
+- Schema patterns: `packages/mcp-core/src/schema.ts`
 - Mock examples: `packages/mcp-server-mocks/src/handlers/`

docs/architecture.md

@@ -206,6 +206,30 @@ Each tool follows a consistent structure:
 - Sentry MCP must stay under 25 tools (target: ~20)
 - Consolidate functionality where possible
 - Consider parameter variants over new tools
+- Long-tail operations can live in the tool catalog and, in experimental mode,
+  be reached through `search_tools` and `execute_tool` instead of being exposed
+  directly in `tools/list`
+
+**Tool Catalog and Direct Exposure:**
+
+Tool modules define the operation itself. Ordinary tools are catalog-eligible by
+default and can be reached through `search_tools` and `execute_tool` when the
+catalog gateway is enabled in experimental mode.
+Ordinary operation modules live as flat files under
+`packages/mcp-core/src/tools/catalog/` and are registered in
+`packages/mcp-core/src/tools/catalog/index.ts`.
+`packages/mcp-core/src/tools/support/` contains helper modules for tools that
+need more implementation structure.
+`packages/mcp-core/src/tools/catalog-runtime/` contains the shared filtering,
+search, schema, and execution helpers. Wrapper/gateway tools such as
+`search_tools`, `execute_tool`, and `use_sentry` live in
+`packages/mcp-core/src/tools/special/`.
+
+`packages/mcp-core/src/tools/surfaces.ts` only centralizes the subset of
+catalog tools that should also be exposed directly through MCP `tools/list`.
+The same availability filters (skills, constraints, experimental mode, and
+required capabilities) apply before either direct registration or catalog
+execution.
 
 ### 6. Error Handling Philosophy
 

docs/claude-code-plugin.md

@@ -73,7 +73,7 @@ This script:
 1. Imports all tools from `packages/mcp-core/src/tools/index.ts`
 2. Imports all skills from `packages/mcp-core/src/skills.ts`
 3. Writes `toolDefinitions.json` and `skillDefinitions.json` to `packages/mcp-core/src/`
-4. Updates `allowedTools` in both `plugins/sentry-mcp/agents/sentry-mcp.md` and `plugins/sentry-mcp-experimental/agents/sentry-mcp.md`
+4. Updates `allowedTools` in both `plugins/sentry-mcp/agents/sentry-mcp.md` and `plugins/sentry-mcp-experimental/agents/sentry-mcp.md`, using the stable direct surface for the stable plugin and the experimental direct surface for the experimental plugin
 
 The script runs automatically as a `prebuild` and `pretest` hook in `packages/mcp-core/package.json`. Run it explicitly after:
 - Adding, removing, or renaming tools

docs/llms/documentation-style-guide.md

@@ -113,7 +113,7 @@ export const ParamOrganizationSlug = z
 - If multiple sections share a name, include a short hint: `("Zod Patterns" in @docs/common-patterns.md)`
 
 ### Code References:
-- Use concrete paths and identifiers: `@packages/mcp-server/src/tools/search-events/index.ts:buildQuery`
+- Use concrete paths and identifiers: `@packages/mcp-core/src/tools/catalog/search-events.ts:buildQuery`
 - Optional line hints for humans: `server.ts:45-52` (agents may ignore)
 - Prefer real implementations over fabricated examples
 

docs/pr-management.md

@@ -287,8 +287,8 @@ This PR implements comprehensive improvements to the search_events tool...
 4. Check that performance remains optimal
 
 ### File Structure Changes
-- src/tools/search-events.ts → src/tools/search-events/handler.ts
-- Added src/tools/search-events/agent.ts for AI logic
+- src/tools/search-events.ts → src/tools/catalog/search-events.ts
+- Added src/tools/support/search-events/agent.ts for AI logic
 - [... detailed file-by-file breakdown ...]
 ```
 

docs/token-cost-tracking.md

@@ -24,18 +24,19 @@ pnpm run measure-tokens
 ```
 📊 MCP Server Token Cost Report
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
-Total Tokens:     9,069
-Tool Count:       19
-Average/Tool:     477
+Total Tokens:     14,068
+Tool Count:       24
+Average/Tool:     586
 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 
 Per-Tool Breakdown:
 
 ┌─────────────────────────────┬────────┬─────────┐
 │ Tool                        │ Tokens │ % Total │
 ├─────────────────────────────┼────────┼─────────┤
-│ search_docs                 │   1036 │   11.4% │
-│ update_issue                │    757 │    8.3% │
+│ search_events               │   1192 │    8.5% │
+│ update_issue                │   1166 │    8.3% │
+│ search_docs                 │    975 │    6.9% │
 ...
 ```
 
@@ -44,19 +45,19 @@ Per-Tool Breakdown:
 # From repository root
 pnpm run measure-tokens -- -o token-stats.json
 
-# Or from mcp-server package
-cd packages/mcp-server
+# Or from mcp-core package
+cd packages/mcp-core
 pnpm run measure-tokens -- -o token-stats.json
 ```
 
 JSON format:
 ```json
 {
-  "total_tokens": 9069,
-  "tool_count": 19,
-  "avg_tokens_per_tool": 477,
+  "total_tokens": 14068,
+  "tool_count": 24,
+  "avg_tokens_per_tool": 586,
   "tools": [
-    {"name": "search_docs", "tokens": 1036, "percentage": 11.4},
+    {"name": "search_events", "tokens": 1192, "percentage": 8.5},
     ...
   ]
 }
@@ -79,9 +80,9 @@ GitHub Actions workflow runs on every PR and push to main:
 
 ## Understanding the Results
 
-**Current baseline (19 tools, excluding use_sentry):**
-- ~9,069 tokens total
-- ~477 tokens/tool average
+**Current baseline (24 tools, excluding use_sentry):**
+- ~14,068 tokens total
+- ~586 tokens/tool average
 
 **Tool count limits:**
 - **Target:** ≤20 tools (current best practice)
@@ -96,7 +97,7 @@ GitHub Actions workflow runs on every PR and push to main:
 
 **Tokenizer:** Uses `tiktoken` with GPT-4's `cl100k_base` encoding (good approximation for Claude).
 
-**Script location:** `packages/mcp-server/scripts/measure-token-cost.ts`
+**Script location:** `packages/mcp-core/scripts/measure-token-cost.ts`
 
 **CLI options:**
 ```bash
@@ -123,6 +124,6 @@ tsx measure-token-cost.ts --help       # Show help
 
 ## References
 
-- Script: `packages/mcp-server/scripts/measure-token-cost.ts`
+- Script: `packages/mcp-core/scripts/measure-token-cost.ts`
 - Workflow: `.github/workflows/token-cost.yml`
 - Tool limits: See "Tool Count Limits" in `docs/adding-tools.md`

packages/mcp-core/scripts/generate-definitions.ts

@@ -16,7 +16,9 @@ const __dirname = path.dirname(__filename);
 
 // Lazy imports of server modules to avoid type bleed
 const toolsModule = await import("../src/tools/index.ts");
+const surfacesModule = await import("../src/tools/surfaces.ts");
 const skillsModule = await import("../src/skills.ts");
+const toolTypesModule = await import("../src/tools/types.ts");
 
 function writeJson(file: string, data: unknown) {
   fs.writeFileSync(file, JSON.stringify(data, null, 2));
@@ -36,13 +38,21 @@ function zodFieldMapToJsonSchema(
 
 // Plugin variants whose agent frontmatter gets synced by this script.
 // Add new entries here when creating a new plugin variant.
-const PLUGIN_AGENT_DIRS = ["sentry-mcp", "sentry-mcp-experimental"];
+const PLUGIN_AGENT_CONFIGS = [
+  { dir: "sentry-mcp", experimentalMode: false },
+  { dir: "sentry-mcp-experimental", experimentalMode: true },
+] as const;
 const PLUGINS_DIR = path.join(__dirname, "../../../plugins");
 
+function agentConfigs() {
+  return PLUGIN_AGENT_CONFIGS.map((config) => ({
+    ...config,
+    path: path.join(PLUGINS_DIR, config.dir, "agents/sentry-mcp.md"),
+  }));
+}
+
 function agentPaths(): string[] {
-  return PLUGIN_AGENT_DIRS.map((dir) =>
-    path.join(PLUGINS_DIR, dir, "agents/sentry-mcp.md"),
-  );
+  return agentConfigs().map((config) => config.path);
 }
 
 function byName<T extends { name: string }>(a: T, b: T) {
@@ -54,7 +64,11 @@ function isNonNull<T>(value: T | null): value is T {
 }
 
 // Tools
-function generateToolDefinitions() {
+function generateToolDefinitions({
+  experimentalMode,
+}: {
+  experimentalMode: boolean;
+}) {
   const toolsDefault = toolsModule.default as
     | Record<string, unknown>
     | undefined;
@@ -70,9 +84,13 @@ function generateToolDefinitions() {
       description: string;
       inputSchema: Record<string, ZodTypeAny>;
       requiredScopes: string[]; // must exist on all tools (can be empty)
-      internalOnly?: boolean;
+      experimental?: boolean;
+      hideInExperimentalMode?: boolean;
     };
-    if (t.internalOnly) {
+    if (!surfacesModule.isDefaultTopLevelToolName(t.name)) {
+      return null;
+    }
+    if (!toolTypesModule.isToolVisibleInMode(t, experimentalMode)) {
       return null;
     }
     if (!Array.isArray(t.requiredScopes)) {
@@ -139,10 +157,12 @@ async function generateSkillDefinitions() {
         description: string;
         skills: string[];
         requiredScopes: string[];
-        internalOnly?: boolean;
       };
 
-      if (t.internalOnly) {
+      if (
+        surfacesModule.isWrapperToolName(t.name) ||
+        surfacesModule.isCatalogInfrastructureToolName(t.name)
+      ) {
         continue;
       }
 
@@ -222,6 +242,7 @@ function isUpToDate(outDir: string): boolean {
   // Check other input files
   const otherInputs = [
     path.join(__dirname, "../src/skills.ts"),
+    path.join(__dirname, "../src/tools/surfaces.ts"),
     path.join(__dirname, "generate-definitions.ts"),
   ];
   for (const inputPath of otherInputs) {
@@ -267,38 +288,33 @@ async function main() {
 
     console.log("Generating tool and skill definitions...");
 
-    const tools = generateToolDefinitions();
+    const tools = generateToolDefinitions({ experimentalMode: false });
+    const experimentalTools = generateToolDefinitions({
+      experimentalMode: true,
+    });
     const skills = await generateSkillDefinitions();
 
     writeJson(path.join(outDir, "toolDefinitions.json"), tools);
     writeJson(path.join(outDir, "skillDefinitions.json"), skills);
 
-    // Sync allowedTools in agent frontmatter
-    // Exclude agent-only tools (e.g., use_sentry) since plugins don't use agent mode
-    const agentOnlyToolNames = new Set(
-      Object.values(
-        toolsModule.default as Record<
-          string,
-          { name: string; agentOnly?: boolean; internalOnly?: boolean }
-        >,
-      )
-        .filter((t) => t.agentOnly || t.internalOnly)
-        .map((t) => t.name),
-    );
-    const toolNames = tools
-      .map((t) => t.name)
-      .filter((name) => !agentOnlyToolNames.has(name));
+    // Sync allowedTools in agent frontmatter with the direct MCP surface for
+    // each plugin's MCP mode.
+    const toolNames = tools.map((t) => t.name);
+    const experimentalToolNames = experimentalTools.map((t) => t.name);
 
     let agentsSynced = 0;
-    for (const agentPath of agentPaths()) {
-      if (fs.existsSync(agentPath)) {
-        syncAgentFrontmatter(agentPath, toolNames);
+    for (const agentConfig of agentConfigs()) {
+      if (fs.existsSync(agentConfig.path)) {
+        syncAgentFrontmatter(
+          agentConfig.path,
+          agentConfig.experimentalMode ? experimentalToolNames : toolNames,
+        );
         agentsSynced++;
       }
     }
 
     console.log(
-      `✅ Generated: tools(${tools.length}), skills(${skills.length}), agents(${agentsSynced})`,
+      `✅ Generated: tools(${tools.length}), experimentalTools(${experimentalTools.length}), skills(${skills.length}), agents(${agentsSynced})`,
     );
   } catch (error) {
     const err = error as Error;