NodeJS robots.txt parser with support for wildcard (*) matching.
-
Updated
Aug 7, 2026 - JavaScript
#
robots-txt
Here are 83 public repositories matching this topic...
Gatsby plugin that automatically creates robots.txt for your site
-
Updated
Jan 29, 2024 - JavaScript
Parser for robots.txt for node.js
-
Updated
Mar 31, 2021 - JavaScript
Generator robots.txt for node js
-
Updated
Jun 19, 2026 - JavaScript
Terminal CLI to download complete web pages (HTML + assets) for offline viewing. Markdown export, robots.txt aware.
nodejs javascript cli npm-package web-scraping robots-txt command-line-tool web-archiving markdown-converter offline-reading wget-alternative html-downloader
-
Updated
May 15, 2026 - JavaScript
A webpack plugin to generate a robots.txt file
-
Updated
Mar 15, 2023 - JavaScript
Smart client-side robots.txt generator for WordPress - security levels, plugin presets, AI crawler blocking, i18n. No backend, no dependencies.
-
Updated
Jun 30, 2026 - JavaScript
🧑🏻👩🏻 "We are people, not machines" - An initiative to know the creators of a website. Contains the information about humans to the web building - A Nuxt Module to statically integrate and generate a humans.txt author file - Based on the HumansTxt Project.
-
Updated
Sep 30, 2021 - JavaScript
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
-
Updated
Jun 23, 2023 - JavaScript
Generate a robots.txt file for your Eleventy site
-
Updated
Oct 17, 2025 - JavaScript
Higher order Next.js config to generate sitemap.xml and robots.txt
-
Updated
Oct 28, 2021 - JavaScript
⚡ JavaScript-aware crawler for security researchers and bug bounty hunters. Extract hidden endpoints and internal subdomains through static and semantic analysis of JS files. Lightweight. Fast. Sneaky.
nodejs javascript sitemap security crawler tool robots-txt infosec web-security bugbounty pentest wayback-machine reconnaissance endpoint-discovery subdomain-enumeration ast-analysis
-
Updated
Jul 22, 2025 - JavaScript
Give AI agents bounded eyes on the public web: crawl, search, and normalize evidence without an unrestricted browser.
nodejs open-source crawler typescript mcp web-crawler web-scraping robots-txt developer-tools browser-automation ai-agents evidence structured-extraction cloudflare-workers ai-agent agent-tools site-mapping llm-ready-data ssrf-protection
-
Updated
Aug 29, 2026 - JavaScript
Generate sitemap and robots.txt for NextJS used web hook from STRAPI
-
Updated
Jan 19, 2021 - JavaScript
A dependency-light Node.js forward HTTP(S) proxy that allows or blocks requests based on the target site's robots.txt.
-
Updated
Jun 22, 2026 - JavaScript
免费开源的 GEO 检测工具 | Free open-source GEO (Generative Engine Optimization) checker — 检测网站能否被 ChatGPT、Claude、Perplexity、Gemini 抓取、理解与引用
seo geo robots-txt ai-search perplexity seo-audit ai-crawler gptbot ai-seo llms-txt generative-engine-optimization claudebot
-
Updated
Aug 21, 2026 - JavaScript
20 free SEO and AEO tools for your terminal plus a curated open-source SEO library: site audit, llms.txt generator and validator, AI crawler checker, pixel-width meta title checker, schema markup generator and validator, broken link checker and keyword clustering.
nodejs sitemap seo robots-txt structured-data schema-markup seo-tools ai-search cli-tools seo-audit technical-seo aeo broken-link-checker llms-txt generative-engine-optimization answer-engine-optimization
-
Updated
Aug 30, 2026 - JavaScript
Generates a robots.txt
-
Updated
Nov 1, 2019 - JavaScript
Chrome extension that audits which paths a site disallows in robots.txt and how many of them Google has indexed anyways.
-
Updated
May 8, 2026 - JavaScript
A robots.txt script for Lambda Edge
-
Updated
Feb 21, 2018 - JavaScript
Improve this page
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."