Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(fetch): add fetching of raw text, pagination and keeping links in the markdown #130

Merged
merged 8 commits into from
Nov 29, 2024
Merged

feat(fetch): add fetching of raw text, pagination and keeping links in the markdown #130

Changes from 1 commit
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
467330d
update fetch server to use readability JS if node is installed
jackadamson Nov 28, 2024
37622d3
add handling of non-html pages
jackadamson Nov 29, 2024
960321f
improve error message to model on fetch failure
jackadamson Nov 29, 2024
e8dcd29
add pagination of fetches so models can avoid reading a full page if …
jackadamson Nov 29, 2024
b6710da
add argument to fetch raw html
jackadamson Nov 29, 2024
5552af1
format with black
jackadamson Nov 29, 2024
c820086
update README to reflect new capabilities
jackadamson Nov 29, 2024
ea42a21
add doc strings for readabilty and constrain types
jackadamson Nov 29, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update README to reflect new capabilities
  • Loading branch information
@jackadamson
jackadamson committed Nov 29, 2024
commit c820086b35e57f1254315d04443f2e04bc4431fd
11 changes: 8 additions & 3 deletions src/fetch/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,21 +2,26 @@

A Model Context Protocol server that provides web content fetching capabilities. This server enables LLMs to retrieve and process content from web pages, converting HTML to markdown for easier consumption.

Presently the server only supports fetching HTML content.
The fetch tool will truncate the response, but by using the `start_index` argument, you can specify where to start the content extraction. This lets models read a webpage in chunks, until they find the information they need.

### Available Tools

- `fetch` - Fetches a URL from the internet and extracts its contents as markdown.
- `url` (string, required): URL to fetch
- `max_length` (integer, optional): Maximum number of characters to return (default: 5000)
- `start_index` (integer, optional): Start content from this character index (default: 0)
- `raw` (boolean, optional): Get raw content without markdown conversion (default: false)

### Prompts

- **fetch**
- Fetch a URL and extract its contents as markdown
- Argument: `url` (string, required): URL to fetch
- Arguments:
- `url` (string, required): URL to fetch

## Installation

Optionally: Install node.js, this will cause the fetch serve to use a different HTML simplifier that is more robust.
Optionally: Install node.js, this will cause the fetch server to use a different HTML simplifier that is more robust.

### Using uv (recommended)

Expand Down