MCP INGEST

🚧 This project is currently under development 🚧

Core implementation of the Model Context Protocol (MCP), designed as an MCP Server for Ingesting APIs, SDKs, and Schemas into AI Models. This module provides a hierarchical structure for managing and routing SDKs and other protocol information.

Conceptual Overview

The main goal of mcp-ingest is to act as a central hub for various data sources, preparing and channeling them for AI model consumption. This is envisioned to be achieved by mounting specialized servers for different protocols.

Project Structure

mcp-ingest/
├── .github/workflows/           # GitHub Actions workflows
├── src/
│   ├── config/                  # Configuration files (settings.toml)
│   ├── prompts/                 # Prompt templates
│   ├── resources/               # Resource definitions
│   ├── router/                  # Routing logic
│   ├── tools/                   # Tool implementations
│   ├── exceptions.py
│   ├── logger.py
│   └── server.py                # Main server application
├── tests/                       # Unit and integration tests
├── .gitignore
├── docker-compose.yml           # Docker Compose configuration
├── Dockerfile                   # Dockerfile for building the image
├── entry_local.sh               # Script for local development startup
├── entry.sh                     # Entrypoint script for Docker container
├── renovate.json                # Renovate bot configuration

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.

Prerequisites

• Python >=3.10
• UV (Python package installer and virtual environment manager)

Installation & Running Locally

1. Clone the repository:

   git clone <repository-url>
   cd mcp-ingest

2. Create a virtual environment and install dependencies using UV: Make sure UV is installed. If not, follow the official installation guide.

   uv sync

   This command reads the pyproject.toml file, creates a virtual environment (if one doesn't exist or if uv is configured to do so), and installs all necessary dependencies listed in pyproject.toml and uv.lock.

3. Configure the application: Copy or symlink src/config/settings.toml to the root directory as settings.toml, or ensure your PYTHONPATH allows the application to find src.config.settings. The entry_local.sh script expects BASE_DIR to be correctly set. Modify it if your project is located elsewhere:

   # In entry_local.sh
   BASE_DIR="/Users/nk/Projects/_/mcp-ingest" # Or your actual project path

4. Run the server locally: Execute the local entry script:

   ./entry_local.sh

   This script will navigate to the project directory, activate the virtual environment (assuming it's in .venv), change to the src directory, and start the Python server.

Docker

You can also build and run the application using Docker and Docker Compose.

1. Build the Docker image:

   docker-compose build

2. Run the application using Docker Compose:

   docker-compose up

   The server will be accessible at http://localhost:8008 (or the port specified in docker-compose.yml).

Configuration

Application configuration is managed via settings.toml. Key settings include:

[ ] TODO

The dynaconf library is used for managing configurations, allowing for flexible setup through environment variables or settings files.

MCP Server Configuration Example

To configure this ingestor with an MCP Server, you might use a configuration similar to the following in the server's settings:

      "mcp-ingest": {
        "type": "stdio",
        "command": "bash",
        "args": [
          "/path/to/your/mcp-ingest/entry_local.sh"
          // OR if running in Docker, the command might differ
          // e.g., pointing to entry.sh within the container context
        ]
      }

Make sure the command and args point to the correct execution script and path within your environment. For local development, entry_local.sh is suitable. For a containerized deployment, entry.sh is typically used.

Advanced: Server Composition (Planned Feature)

This project plans to leverage FastMCP's server composition capabilities to create a modular and extensible ingest system. The core idea is to have a central MCP Ingest Server that can dynamically mount various specialized protocol servers (e.g., an OpenAPI server, an SDK documentation server, a schema definition server).

What is Server Composition?

As your MCP applications grow, you might want to organize your tools, resources, and prompts into logical modules or reuse existing server components. FastMCP supports composition through two methods:

1. import_server (Static Composition):

   • How it works: Performs a one-time copy of all components (tools, resources, templates, prompts) from a subserver into the main server. A prefix is added to avoid naming conflicts (e.g., subserver.tool(name="my_tool") becomes main_mcp.tool(name="{prefix}_my_tool")).
   • Updates: Changes made to the subserver after importing are not reflected in main_mcp. The subserver’s lifespan context is also not executed by the main server.
   • Best for: Bundling finalized, stable components.

2. mount (Dynamic Composition / Live Linking):

   • How it works: Creates a live link between the main_mcp server and the subserver. Instead of copying components, requests for components matching the prefix are delegated to the subserver at runtime.
   • Updates: Changes to the mounted server are immediately reflected when accessed through the parent.
   • Best for: Modular runtime composition, allowing dynamic updates. This is the planned approach for mcp-ingest.

Why Use Server Composition?

(Based on content from FastMCP Documentation)

• Modularity: Break down large applications into smaller, focused servers (e.g., a WeatherServer, a DatabaseServer). For mcp-ingest, this could mean an OpenAPIServer, SDKParserServer, etc.
• Reusability: Create common utility servers and mount them wherever needed.
• Teamwork: Different teams can work on separate FastMCP servers that are later combined.
• Organization: Keep related functionality grouped together logically.

How Mounting Works (Simplified)

When main_mcp.mount("prefix", subserver) is called:

• A live link is established between the parent server and the mounted server.
• The parent server uses the specified "prefix" to route requests to the subserver.
• Requests for components matching the prefix (e.g., prefix_tool_name) are delegated to the subserver at runtime.
• FastMCP automatically handles the prefixing for tools, resources, templates, and prompts to avoid naming collisions, similar to import_server.

FastMCP also supports different mounting modes (Direct vs. Proxy Mounting) and resource prefix formats (Path Format vs. Protocol Format), offering flexibility for various scenarios. Direct mounting is the default, accessing objects in memory, while proxy mounting treats the mounted server as a separate entity, preserving its full client lifecycle and lifespan context. The resource prefix format can be configured globally or per-server, with "path format" (resource://prefix/path/to/resource) being the default and recommended since FastMCP 2.4.

Note: The implementation of server mounting in mcp-ingest is a planned future enhancement and is not yet available in the current version.

License

This project is licensed under the MIT License - see the LICENSE file for details.