A list of useful Open Source tools and scrapers to gather data for LLMs:
| Name | |
|---|---|
| gitingest | Replace hub with ingest in any github url to get a prompt-friendly extract of a codebase |
| repomix | Packs your entire repository into a single, AI-friendly file |
| llm-scraper | Turn any webpage into structured data using LLMs |
| crawl4ai | LLM friendly web crawler & scraper |
| trafilatura | Python & Command-line tool to gather text and metadata on the web |
| RepoToTextForLLMs | Simple Python script to fetch repo content |
| marker | Convert PDF to markdown or JSON quickly |
| reader | Convert any URL to an LLM-friendly input with a simple prefix https://r.jina.ai/ |
| files-to-prompt | Concatenate a directory full of files into a single prompt for use with LLMs |
| docling | Simplifies document processing and parsing of diverse formats |
| firecrawl | API to turn websites into LLM-ready markdown or structured data, can be self-hosted (with limitations) |
| llmstxt-generator | API to generate llms.txtfiles from websites for LLM training and inference |
- https://github.com/mlabonne/llm-datasets: Curated list of datasets and tools specifically for post-training.