Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 11 additions & 20 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ The Azure Functions API leverages the readability library and BeautifulSoup to e

### MCP Server

The Model Context Protocol (MCP) server is a FastAPI-based implementation that provides web search capabilities with enhanced content extraction. It follows the MCP specification for standardized AI model interactions.
The Model Context Protocol (MCP) server is a FastAPI-based implementation that provides web search capabilities with enhanced content extraction. It follows the MCP specification for standardized AI model interactions and uses SSE transport for compatibility with LiteLLM and other MCP clients.

## Features
- **Content Extraction**: Utilizes the readability library for clean text extraction
Expand All @@ -38,22 +38,22 @@ The Model Context Protocol (MCP) server is a FastAPI-based implementation that p
- See the `customgpt` directory for specific documentation

### MCP Server (Docker)
- Current version: 2.1.0
- Docker image: `tmfrisinger/webcat:2.1.0` or `tmfrisinger/webcat:latest`
- Current version: 2.2.0 (simplified - no authentication required)
- Docker image: `tmfrisinger/webcat:2.2.0` or `tmfrisinger/webcat:latest`

#### Running with Docker
```bash
# Run with Serper API (recommended for best results)
docker run -p 8000:8000 -e SERPER_API_KEY=your_key -e WEBCAT_API_KEY=your_api_key tmfrisinger/webcat:latest
docker run -p 8000:8000 -e SERPER_API_KEY=your_key tmfrisinger/webcat:2.2.0

# Run with free DuckDuckGo fallback (requires WEBCAT_API_KEY for authentication)
docker run -p 8000:8000 -e WEBCAT_API_KEY=your_api_key tmfrisinger/webcat:latest
# Run with free DuckDuckGo fallback (no API key required)
docker run -p 8000:8000 tmfrisinger/webcat:2.2.0

# Run on a custom port
docker run -p 9000:9000 -e PORT=9000 -e SERPER_API_KEY=your_key -e WEBCAT_API_KEY=your_api_key tmfrisinger/webcat:latest
docker run -p 9000:9000 -e PORT=9000 -e SERPER_API_KEY=your_key tmfrisinger/webcat:2.2.0

# With custom rate limiting
docker run -p 8000:8000 -e SERPER_API_KEY=your_key -e WEBCAT_API_KEY=your_api_key -e RATE_LIMIT_WINDOW=60 -e RATE_LIMIT_MAX_REQUESTS=10 tmfrisinger/webcat:latest
docker run -p 8000:8000 -e SERPER_API_KEY=your_key -e RATE_LIMIT_WINDOW=60 -e RATE_LIMIT_MAX_REQUESTS=10 tmfrisinger/webcat:2.2.0
```

#### Building the Docker Image
Expand All @@ -70,23 +70,11 @@ For more detailed Docker information, see the `docker/README.md` file.
## Configuration

### Environment Variables
- `WEBCAT_API_KEY`: **Required** - Your custom API key for authentication (user-generated security token)
- `SERPER_API_KEY`: Your Serper API key (optional, enables premium search results)
- `PORT`: The port to run the server on (default: 8000)
- `RATE_LIMIT_WINDOW`: Time window in seconds for rate limiting (default: 60)
- `RATE_LIMIT_MAX_REQUESTS`: Max requests per window (default: 10)

#### Generating WEBCAT_API_KEY
The `WEBCAT_API_KEY` is a user-generated security token that you create to protect your WebCat server. You can generate one using:

```bash
# Generate a secure random key
export WEBCAT_API_KEY="sk-webcat-$(openssl rand -hex 32)"

# Or create your own custom key
export WEBCAT_API_KEY="sk-webcat-my-secure-key-2024"
```

## Testing

The project includes comprehensive test suites for both the Azure Functions API and the MCP Server:
Expand All @@ -108,6 +96,9 @@ python -m unittest test_mcp_server.py
- **Serper API**: Premium search results with high accuracy and comprehensive coverage (requires API key)
- **DuckDuckGo Fallback**: Free search functionality with good quality results (no API key required)

## LiteLLM Compatibility
WebCat's MCP server now uses SSE (Server-Sent Events) transport instead of streamable-http, making it fully compatible with LiteLLM and other MCP clients that expect SSE protocol. No authentication is required, making it simple to integrate.

## Contributing

Contributions are welcome! Please feel free to submit a Pull Request.
Expand Down
43 changes: 11 additions & 32 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,29 +26,18 @@ This directory contains the **FastMCP-based Model Context Protocol (MCP) server*
# Build the image
./build.sh

# Generate a secure API key (choose one method)
export WEBCAT_API_KEY="sk-webcat-$(openssl rand -hex 32)"
# OR use your own secure string
export WEBCAT_API_KEY="sk-webcat-my-secure-key-2024"

# Run with free DuckDuckGo fallback (no external API key needed)
docker run -p 8000:8000 \
-e WEBCAT_API_KEY="$WEBCAT_API_KEY" \
webcat:latest
# Run with free DuckDuckGo fallback (no API keys needed)
docker run -p 8000:8000 webcat:latest

# Or run with premium Serper API
docker run -p 8000:8000 \
-e WEBCAT_API_KEY="$WEBCAT_API_KEY" \
-e SERPER_API_KEY=your_serper_api_key \
webcat:latest
```

### 2. Using Docker Compose

```bash
# Generate your secure API key
export WEBCAT_API_KEY="sk-webcat-$(openssl rand -hex 32)"

# Optionally set Serper API key for premium search
export SERPER_API_KEY=your_serper_api_key # Optional

Expand All @@ -60,27 +49,19 @@ docker-compose up

### Environment Variables

- `WEBCAT_API_KEY`: **Required** - Your custom API key for authentication (user-generated security token)
- `SERPER_API_KEY`: **Optional** - Serper API key for premium search (falls back to DuckDuckGo if not set)
- `PORT`: Port to run the server on (default: 8000)
- `LOG_LEVEL`: Logging level (default: INFO)
- `LOG_DIR`: Directory for log files (default: /tmp)

### API Key Security
### Simplified Setup

The `WEBCAT_API_KEY` is a **user-generated security token** that you create to protect your WebCat server:
WebCat now runs without authentication requirements, making it easier to integrate:

- 🔐 **You generate it** - Create any secure string (e.g., `sk-webcat-your-secret-key-123`)
- 🛡️ **Authentication layer** - Prevents unauthorized access to your search server
- 🔑 **Required for all requests** - MCP clients must provide this key to use the server
- 💡 **Best practices**: Use a long, random string with prefixes like `sk-webcat-` for clarity

Example secure keys:
```bash
export WEBCAT_API_KEY="sk-webcat-$(openssl rand -hex 32)"
export WEBCAT_API_KEY="webcat-prod-$(date +%s)-$(openssl rand -hex 16)"
export WEBCAT_API_KEY="sk-webcat-my-secure-key-2024"
```
- 🚀 **No API key required** - Simply run the container and start using
- 🔓 **Open access** - Perfect for development and trusted environments
- ⚡ **Quick setup** - Get started in seconds without key generation
- 🔧 **Easy integration** - Works seamlessly with any MCP client

## MCP Protocol Endpoints

Expand Down Expand Up @@ -157,7 +138,7 @@ python -m pytest -v

### Method 3: Test MCP Protocol Directly

**⚠️ Note**: `streamable-http` requires proper MCP initialization flow. Here's the correct sequence:
**⚠️ Note**: The server now uses SSE transport for better LiteLLM compatibility. Here's the correct sequence:

#### Step 1: Initialize MCP Session
```bash
Expand Down Expand Up @@ -204,9 +185,7 @@ Use with Claude Desktop or other MCP-compatible clients:
"mcpServers": {
"webcat": {
"command": "docker",
"args": ["run", "-i", "--rm",
"-e", "WEBCAT_API_KEY=your_key",
"webcat:latest"]
"args": ["run", "-i", "--rm", "webcat:latest"]
}
}
}
Expand Down Expand Up @@ -348,7 +327,7 @@ The server acts as an MCP-compliant bridge between AI models and web search capa
- **`tests/test_mcp_server.py`** - Content processing and utility functions

### **Integration Tests (Require Running Services) ✅**
- **`test_mcp_protocol.py`** - Complete MCP streamable-http protocol flow
- **`test_mcp_protocol.py`** - Complete MCP SSE protocol flow
- **`test_duckduckgo_fallback.py`** - Full server integration with DuckDuckGo

### **CI/CD Strategy:**
Expand Down
8 changes: 4 additions & 4 deletions docker/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ set -e
# Configuration
IMAGE_NAME="webcat"
USER="tmfrisinger"
VERSION=2.1.0
VERSION=2.2.2
TAG="${USER}/${IMAGE_NAME}:${VERSION}"
LATEST="${USER}/${IMAGE_NAME}:latest"
DEFAULT_PORT=8000
Expand All @@ -17,13 +17,13 @@ docker build -t ${TAG} -t ${LATEST} -f docker/Dockerfile .

echo "Docker image built successfully!"
echo "To run the Docker image locally:"
echo "docker run -p ${DEFAULT_PORT}:${DEFAULT_PORT} -e SERPER_API_KEY=your_key -e WEBCAT_API_KEY=your_api_key ${LATEST}"
echo "docker run -p ${DEFAULT_PORT}:${DEFAULT_PORT} -e SERPER_API_KEY=your_key ${LATEST}"
echo ""
echo "To run on a custom port:"
echo "docker run -p 9000:9000 -e PORT=9000 -e SERPER_API_KEY=your_key -e WEBCAT_API_KEY=your_api_key ${LATEST}"
echo "docker run -p 9000:9000 -e PORT=9000 -e SERPER_API_KEY=your_key ${LATEST}"
echo ""
echo "To customize logging:"
echo "docker run -p ${DEFAULT_PORT}:${DEFAULT_PORT} -e SERPER_API_KEY=your_key -e WEBCAT_API_KEY=your_api_key -e LOG_LEVEL=DEBUG ${LATEST}"
echo "docker run -p ${DEFAULT_PORT}:${DEFAULT_PORT} -e SERPER_API_KEY=your_key -e LOG_LEVEL=DEBUG ${LATEST}"
echo ""
echo "To push to a registry, run:"
echo "docker push ${TAG}"
Expand Down
6 changes: 2 additions & 4 deletions docker/debug_search.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,10 +16,8 @@ def debug_search():
# Get API key from environment
api_key = os.environ.get("SERPER_API_KEY", "")
if not api_key:
api_key = os.environ.get("WEBCAT_API_KEY", "")

if not api_key:
print("❌ No API key found in environment variables.")
print("❌ No Serper API key found in environment variables.")
print("💡 Set SERPER_API_KEY to use premium search, or use DuckDuckGo fallback instead.")
return

# Mask the key for display
Expand Down
1 change: 0 additions & 1 deletion docker/docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,6 @@ services:
- "${PORT:-8000}:${PORT:-8000}"
environment:
- SERPER_API_KEY=${SERPER_API_KEY}
- WEBCAT_API_KEY=${WEBCAT_API_KEY}
- PORT=${PORT:-8000}
- LOG_LEVEL=${LOG_LEVEL:-INFO}
- LOG_DIR=/var/log/webcat
Expand Down
21 changes: 4 additions & 17 deletions docker/mcp_server.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,24 +68,11 @@ class SearchResult(BaseModel):

# Configure API keys
SERPER_API_KEY = os.environ.get("SERPER_API_KEY", "")
WEBCAT_API_KEY = os.environ.get("WEBCAT_API_KEY", "")

# Improved logging for API keys
if WEBCAT_API_KEY:
key_length = len(WEBCAT_API_KEY)
masked_key = WEBCAT_API_KEY[:4] + "*" * (key_length - 8) + WEBCAT_API_KEY[-4:] if key_length > 8 else "****"
logging.info(f"WEBCAT_API_KEY is set (masked: {masked_key})")
else:
logging.warning("WEBCAT_API_KEY is not set! Authentication will not work.")

logging.info(f"Using SERPER API key from environment: {'Set' if SERPER_API_KEY else 'Not set'}")

# Create FastMCP instance
mcp_server = FastMCP(
name="WebCat Search",
description="A server providing web search capabilities to models following the MCP protocol",
authentication_key=WEBCAT_API_KEY
)
# Create FastMCP instance (no authentication required)
mcp_server = FastMCP("WebCat Search")

# Utility functions
def fetch_search_results(query: str, api_key: str) -> List[Dict[str, Any]]:
Expand Down Expand Up @@ -364,9 +351,9 @@ async def health_check():
port = int(os.environ.get("PORT", 8000))
logging.info(f"Starting FastMCP server on port {port}")

# Run the server with proper streamable-http configuration
# Run the server with SSE transport for LiteLLM compatibility
mcp_server.run(
transport="streamable-http",
transport="sse",
host="0.0.0.0",
port=port,
path="/mcp" # Explicit path for MCP endpoint
Expand Down
2 changes: 1 addition & 1 deletion docker/test_duckduckgo_fallback.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@

# Set up minimal environment variables for testing
os.environ["LOG_DIR"] = tempfile.gettempdir()
os.environ["WEBCAT_API_KEY"] = "test_key_for_testing"
# No API key needed for testing

# Set up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
Expand Down
8 changes: 4 additions & 4 deletions docker/test_mcp_protocol.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
"""Test script for the MCP streamable-http protocol."""
"""Test script for the MCP SSE protocol."""

import requests
import json
Expand All @@ -10,7 +10,7 @@

@pytest.mark.integration
def test_mcp_protocol():
"""Test the MCP streamable-http protocol.
"""Test the MCP SSE protocol.

This is an integration test that requires a running MCP server.
"""
Expand Down Expand Up @@ -163,7 +163,7 @@ def check_server_health():
return False

if __name__ == "__main__":
parser = argparse.ArgumentParser(description='Test MCP streamable-http protocol')
parser = argparse.ArgumentParser(description='Test MCP SSE protocol')
parser.add_argument('--check-health', action='store_true',
help="Only check if server is running")
args = parser.parse_args()
Expand All @@ -173,7 +173,7 @@ def check_server_health():
else:
if not check_server_health():
print("\n💡 Start the server first:")
print(" docker run -d -p 8000:8000 -e WEBCAT_API_KEY='test-key' tmfrisinger/webcat:latest")
print(" docker run -d -p 8000:8000 tmfrisinger/webcat:latest")
exit(1)

# Run the pytest test directly
Expand Down
2 changes: 1 addition & 1 deletion docker/test_search_functions.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

# Set up minimal environment variables for testing
os.environ["LOG_DIR"] = tempfile.gettempdir()
os.environ["WEBCAT_API_KEY"] = "test_key_for_testing"
# No API key needed for testing

# Set up logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
Expand Down