NodeJS robots.txt parser with support for wildcard (*) matching.
-
Updated
Oct 28, 2024 - JavaScript
#
robots-txt
Here are 33 public repositories matching this topic...
Gatsby plugin that automatically creates robots.txt for your site
-
Updated
Jan 29, 2024 - JavaScript
Parser for robots.txt for node.js
-
Updated
Mar 31, 2021 - JavaScript
Generator robots.txt for node js
-
Updated
Jan 4, 2023 - JavaScript
A webpack plugin to generate a robots.txt file
-
Updated
Mar 15, 2023 - JavaScript
🧑🏻👩🏻 "We are people, not machines" - An initiative to know the creators of a website. Contains the information about humans to the web building - A Nuxt Module to statically integrate and generate a humans.txt author file - Based on the HumansTxt Project.
-
Updated
Sep 30, 2021 - JavaScript
A lightweight robots.txt parser for Node.js with support for wildcards, caching and promises.
-
Updated
Jun 23, 2023 - JavaScript
Higher order Next.js config to generate sitemap.xml and robots.txt
-
Updated
Oct 28, 2021 - JavaScript
Generate a robots.txt file for your Eleventy site
-
Updated
Aug 1, 2024 - JavaScript
Generate sitemap and robots.txt for NextJS used web hook from STRAPI
-
Updated
Jan 19, 2021 - JavaScript
Generates a robots.txt
-
Updated
Nov 1, 2019 - JavaScript
A robots.txt script for Lambda Edge
-
Updated
Feb 21, 2018 - JavaScript
⚡ JavaScript-aware crawler for security researchers and bug bounty hunters. Extract hidden endpoints and internal subdomains through static and semantic analysis of JS files. Lightweight. Fast. Sneaky.
nodejs javascript sitemap security crawler tool robots-txt infosec web-security bugbounty pentest wayback-machine reconnaissance endpoint-discovery subdomain-enumeration ast-analysis
-
Updated
Apr 18, 2025 - JavaScript
🤖 Robots.txt generator done right.
-
Updated
Mar 9, 2017 - JavaScript
🤖 Handle and parse a site's robots.txt file and extract actionable information
-
Updated
Dec 4, 2017 - JavaScript
nodejs web crawler
nodejs javascript npm sitemap crawler node web yarn es6 npm-package robots-txt npm-module depth nodejs-web-crawler
-
Updated
Feb 24, 2017 - JavaScript
🤖 Browser extension to check for and preview a site's robots.txt in a new tab (if it exists)
-
Updated
Oct 4, 2018 - JavaScript
Chrome extension which blocks urls based on robots.txt (compatible to Chrome 41)
-
Updated
Oct 9, 2018 - JavaScript
Front-end workflow to start a new project with Eleventy and Webpack.
stylelint static-site-generator template boilerplate performance webpack postcss eslint service-worker accessibility minify nunjucks seo robots-txt htaccess posthtml imagemin eleventy terser sanitize-css
-
Updated
Oct 27, 2020 - JavaScript
A tool for debugging robots.txt
-
Updated
Mar 23, 2018 - JavaScript
Improve this page
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."