Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ README.md
*.md

# Build artifacts
fetch-mcp-server
gofetch
*.exe
*.dll
*.so
Expand Down
2 changes: 1 addition & 1 deletion .golangci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ formatters:
sections:
- standard
- default
- prefix(github.com/StacklokLabs/fetch)
- prefix(github.com/StacklokLabs/gofetch)
exclusions:
generated: lax
paths:
Expand Down
6 changes: 3 additions & 3 deletions .ko.yaml
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
builds:
- id: fetch-server
- id: gofetch-server
dir: .
main: ./cmd/server
ldflags:
- -X main.version={{.Env.VERSION}}
labels:
org.opencontainers.image.created: "{{.Env.CREATION_TIME}}"
org.opencontainers.image.description: "fetch - A HTTP fetching MCP server."
org.opencontainers.image.description: "gofetch - A HTTP fetching MCP server."
org.opencontainers.image.licenses: "Apache-2.0"
org.opencontainers.image.revision: "{{.Env.GITHUB_SHA}}"
org.opencontainers.image.source: "{{.Env.GITHUB_SERVER_URL}}/{{.Env.GITHUB_REPOSITORY}}"
org.opencontainers.image.title: "fetch"
org.opencontainers.image.title: "gofetch"
org.opencontainers.image.url: "{{.Env.GITHUB_SERVER_URL}}/{{.Env.GITHUB_REPOSITORY}}"
org.opencontainers.image.version: "{{.Env.VERSION}}"
22 changes: 11 additions & 11 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# Contributing to `fetch` <!-- omit from toc -->
# Contributing to `gofetch` <!-- omit from toc -->

First off, thank you for taking the time to contribute to fetch! :+1: :tada:
fetch is released under the Apache 2.0 license. If you would like to contribute
First off, thank you for taking the time to contribute to gofetch! :+1: :tada:
gofetch is released under the Apache 2.0 license. If you would like to contribute
something or want to hack on the code, this document should help you get
started. You can find some hints for starting development in fetch's
[README](https://github.com/StacklokLabs/fetch/blob/main/README.md).
started. You can find some hints for starting development in gofetch's
[README](https://github.com/StacklokLabs/gofetch/blob/main/README.md).

## Table of contents <!-- omit from toc -->

Expand All @@ -19,17 +19,17 @@ started. You can find some hints for starting development in fetch's
## Code of conduct

This project adheres to the
[Contributor Covenant](https://github.com/StacklokLabs/fetch/blob/main/CODE_OF_CONDUCT.md)
[Contributor Covenant](https://github.com/StacklokLabs/gofetch/blob/main/CODE_OF_CONDUCT.md)
code of conduct. By participating, you are expected to uphold this code. Please
report unacceptable behavior to
[code-of-conduct@stacklok.dev](mailto:code-of-conduct@stacklok.dev).

## Reporting security vulnerabilities

If you think you have found a security vulnerability in fetch please DO NOT
If you think you have found a security vulnerability in gofetch please DO NOT
disclose it publicly until we've had a chance to fix it. Please don't report
security vulnerabilities using GitHub issues; instead, please follow this
[process](https://github.com/StacklokLabs/fetch/blob/main/SECURITY.md)
[process](https://github.com/StacklokLabs/gofetch/blob/main/SECURITY.md)

## How to contribute

Expand All @@ -46,7 +46,7 @@ sample project that reproduces the problem.
### Not sure how to start contributing?

PRs to resolve existing issues are greatly appreciated and issues labeled as
["good first issue"](https://github.com/StacklokLabs/fetch/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)
["good first issue"](https://github.com/StacklokLabs/gofetch/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22)
are a great place to start!

### Pull request process
Expand All @@ -56,14 +56,14 @@ message to indicate that the contributor agrees to the Developer Certificate of
Origin. For additional details, check out the [DCO instructions](dco.md).

- Create an issue outlining the fix or feature.
- Fork the fetch repository to your own GitHub account and clone it locally.
- Fork the gofetch repository to your own GitHub account and clone it locally.
- Hack on your changes.
- Correctly format your commit messages, see
[Commit message guidelines](#commit-message-guidelines) below.
- Open a PR by ensuring the title and its description reflect the content of the
PR.
- Ensure that CI passes, if it fails, fix the failures.
- Every pull request requires a review from the core fetch team before merging.
- Every pull request requires a review from the core gofetch team before merging.
- Once approved, all of your commits will be squashed into a single commit with
your PR title.

Expand Down
81 changes: 53 additions & 28 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,23 @@
# Fetch MCP Server

An SSE Go implementation of the `fetch` MCP server that retrieves web content.
<div align="center">
<img src="docs/gofetch.png" width="300">

[![GitHub Release][release-img]][release]
[![Test][test-img]][test]
[![Go Report Card][go-report-img]][go-report]
[![License: Apache-2.0][license-img]][license]
[![GitHub Downloads][github-downloads-img]][release]
![Docker Pulls][docker-pulls]
</div>

<!-- <img src="docs/gofetch.png" alt="gofetch" height="300" style="display: inline-block; vertical-align: top; margin-right: 20px;"><span style="display: inline-block; vertical-align: top;"> -->

An SSE Go implementation of the `gofetch` MCP server that retrieves web content.</span>


## Features

- **Web Content Retrieval**: Fetches URLs and extracts textual content
- **Web Content Retrieval**: gofetches URLs and extracts textual content
- **Content Extraction**: Extract main content from web pages
- **Robots.txt Compliance**: Respects robots.txt rules (can be disabled)
- **Configurable**: Supports custom user agents and proxy settings
Expand Down Expand Up @@ -38,8 +51,8 @@ but has the following benefits:
1. Clone the repository:

```bash
git clone https://github.com/StacklokLabs/fetch.git
cd fetch
git clone https://github.com/StacklokLabs/gofetch.git
cd gofetch
```

2. Install dependencies:
Expand Down Expand Up @@ -74,40 +87,40 @@ The server will start and expose:
- `--addr`: Address to listen on (default: ":8080", can be set via MCP_PORT env
var)
- `--user-agent`: Custom User-Agent string (default: "Mozilla/5.0 (compatible;
MCPFetchBot/1.0)")
MCPGoFetchBot/1.0)")
- `--ignore-robots-txt`: Ignore robots.txt rules
- `--proxy-url`: Proxy URL for requests

#### Examples

```bash
# Basic server on port 8080
./fetch-mcp-server --addr :8080
./gofetch --addr :8080

# Custom port with user agent
./fetch-mcp-server --addr :3000 --user-agent "MyBot/1.0"
./gofetch --addr :3000 --user-agent "MyBot/1.0"

# Ignore robots.txt on custom port
./fetch-mcp-server --addr :8080 --ignore-robots-txt
./gofetch --addr :8080 --ignore-robots-txt

# Use proxy
./fetch-mcp-server --addr :8080 --proxy-url "http://proxy.example.com:8080"
./gofetch --addr :8080 --proxy-url "http://proxy.example.com:8080"

# Use environment variable for port
MCP_PORT=9090 ./fetch-mcp-server
MCP_PORT=9090 ./gofetc
```

### Docker Usage

```bash
# Build Docker image
docker build -t fetch-mcp-server .
docker build -t gofetch .

# Run with default settings
docker run -p 8080:8080 fetch-mcp-server
docker run -p 8080:8080 gofetch

# Run with custom arguments
docker run -p 9090:9090 fetch-mcp-server --addr :9090
docker run -p 9090:9090 gofetch --addr :9090
```

### Testing the Server
Expand All @@ -124,7 +137,7 @@ curl -N http://localhost:8080/sse

## MCP Tools

The server provides a single tool called `fetch` with the following parameters:
The server provides a single tool called `gofetch` with the following parameters:

### Tool: `fetch`

Expand Down Expand Up @@ -191,7 +204,7 @@ task deps

## Running as an MCP Server with ToolHive

fetch can be run as a Model Context Protocol (MCP) server using
gofetch can be run as a Model Context Protocol (MCP) server using
[ToolHive](https://github.com/stacklok/toolhive), which simplifies the
deployment and management of MCP servers.

Expand All @@ -203,21 +216,21 @@ deployment and management of MCP servers.

### Running fetch with ToolHive (Recommended)

The easiest way to run fetch is using the packaged version available in
The easiest way to run gofetch is using the packaged version available in
ToolHive's registry:

```bash
# Register a supported client so ToolHive can auto-configure your environment
thv client setup

# Run the fetch server
thv run fetch --transport sse
thv run gofetch --transport sse

# List running servers
thv list

# Get detailed information about the server
thv registry info fetch
thv registry info gofetch
```

### Advanced Usage with Custom Configuration
Expand All @@ -227,7 +240,7 @@ the container image directly:

```bash
# Run the fetch server using the published container image
thv run --name fetch --transport sse --target-port 8080 ghcr.io/stackloklabs/fetch/server:latest
thv run --name gofetch --transport sse --target-port 8080 ghcr.io/stackloklabs/gofetch/server:latest
```

This command:
Expand All @@ -239,32 +252,32 @@ This command:
To use a specific version instead of the latest:

```bash
thv run --name fetch --transport sse --target-port 8080 ghcr.io/stackloklabs/fetch/server:v0.0.1
thv run --name gofetch --transport sse --target-port 8080 ghcr.io/stackloklabs/gofetch/server:v0.0.1
```

### Managing the fetch Server
### Managing the gofetch Server

To verify that the fetch server is running:
To verify that the gofetch server is running:

```bash
thv list
```

This will show all running MCP servers managed by ToolHive, including the fetch
This will show all running MCP servers managed by ToolHive, including the gofetch
server.

To stop the fetch server:
To stop the gofetch server:

```bash
# For custom named version
thv stop fetch
thv stop gofetch
```

To remove the server instance completely:

```bash
# For custom named version
thv rm fetch
thv rm gofetch
```

## Contributing
Expand All @@ -274,11 +287,23 @@ review the [CONTRIBUTING guide](./CONTRIBUTING.md) for details on how to get
started.

If you run into a bug or have a feature request, please
[open an issue](https://github.com/StacklokLabs/fetch/issues) in the repository
[open an issue](https://github.com/StacklokLabs/gofetch/issues) in the repository
or join us in the `#mcp-servers` channel on our
[community Discord server](https://discord.gg/stacklok).

## License

This project is licensed under the Apache v2 License - see the LICENSE file for
details.

<!-- Badge Links -->
[release-img]: https://img.shields.io/github/release/StacklokLabs/gofetch.svg
[release]: https://github.com/StacklokLabs/gofetch/releases
[test-img]: https://github.com/StacklokLabs/gofetch/workflows/Main%20build/badge.svg
[test]: https://github.com/StacklokLabs/gofetch/actions?query=workflow%3ATest
[go-report-img]: https://goreportcard.com/badge/github.com/StacklokLabs/gofetch
[go-report]: https://goreportcard.com/report/github.com/StacklokLabs/gofetch
[license-img]: https://img.shields.io/badge/License-Apache%202.0-blue.svg
[license]: https://opensource.org/licenses/Apache-2.0
[github-downloads-img]: https://img.shields.io/github/downloads/StacklokLabs/gofetch/total.svg
[docker-pulls]: https://img.shields.io/docker/pulls/stacklok/gofetch.svg
4 changes: 2 additions & 2 deletions Taskfile.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
version: '3'

vars:
BINARY_NAME: fetch-server
BINARY_NAME: gofetch
BUILD_DIR: build
MAIN_PACKAGE: ./cmd/server

Expand Down Expand Up @@ -62,7 +62,7 @@ tasks:
build-image:
desc: Build the image locally with ko
env:
KO_DOCKER_REPO: ghcr.io/stackloklabs/fetch/server
KO_DOCKER_REPO: ghcr.io/stackloklabs/gofetch/server
VERSION: dev-local
cmds:
- ko build --local --bare ./cmd/server
4 changes: 2 additions & 2 deletions cmd/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ import (
"context"
"log"

"github.com/chrisburns/fetch-mcp-server/pkg/config"
"github.com/chrisburns/fetch-mcp-server/pkg/server"
"github.com/stackloklabs/gofetch/pkg/config"
"github.com/stackloklabs/gofetch/pkg/server"
)

func main() {
Expand Down
Binary file added docs/gofetch.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
module github.com/chrisburns/fetch-mcp-server
module github.com/stackloklabs/gofetch

go 1.23.2

Expand Down
4 changes: 2 additions & 2 deletions pkg/fetcher/fetcher.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import (
"net/http"
"strings"

"github.com/chrisburns/fetch-mcp-server/pkg/processor"
"github.com/chrisburns/fetch-mcp-server/pkg/robots"
"github.com/stackloklabs/gofetch/pkg/processor"
"github.com/stackloklabs/gofetch/pkg/robots"
)

// HTTPFetcher handles HTTP requests and content retrieval
Expand Down
4 changes: 2 additions & 2 deletions pkg/fetcher/fetcher_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ import (
"testing"
"time"

"github.com/chrisburns/fetch-mcp-server/pkg/processor"
"github.com/chrisburns/fetch-mcp-server/pkg/robots"
"github.com/stackloklabs/gofetch/pkg/processor"
"github.com/stackloklabs/gofetch/pkg/robots"
)

// createMockServer creates a test HTTP server with various endpoints
Expand Down
12 changes: 6 additions & 6 deletions pkg/server/server.go
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Package server provides the MCP server implementation for fetching web content.
// Package server provides the MCP server implementation for gofetching web content.
package server

import (
Expand All @@ -9,12 +9,12 @@ import (
"net/url"
"time"

"github.com/chrisburns/fetch-mcp-server/pkg/config"
"github.com/chrisburns/fetch-mcp-server/pkg/fetcher"
"github.com/chrisburns/fetch-mcp-server/pkg/processor"
"github.com/chrisburns/fetch-mcp-server/pkg/robots"
"github.com/mark3labs/mcp-go/mcp"
"github.com/mark3labs/mcp-go/server"
"github.com/stackloklabs/gofetch/pkg/config"
"github.com/stackloklabs/gofetch/pkg/fetcher"
"github.com/stackloklabs/gofetch/pkg/processor"
"github.com/stackloklabs/gofetch/pkg/robots"
)

// FetchServer represents the MCP server for fetching web content
Expand Down Expand Up @@ -142,7 +142,7 @@ func (fs *FetchServer) Start() error {

// logServerStartup prints startup information
func (fs *FetchServer) logServerStartup() {
log.Printf("=== Starting MCP Fetch Server ===")
log.Printf("=== Starting MCP gofetch Server ===")
log.Printf("Server address: %s", fs.config.Address)
log.Printf("User agent: %s", fs.config.UserAgent)
log.Printf("Ignore robots.txt: %v", fs.config.IgnoreRobots)
Expand Down
Loading