robots-exclusion-protocol

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

Generate an XML sitemap for a GitHub Pages site using GitHub Actions

Manage the robots.txt from the Kirby config file

Alternative robots parser module for Python

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R

Robots for Kirby CMS

Chrome extension that audits which paths a site disallows in robots.txt and how many of them Google has indexed anyways.

RFC 9309 spec compliant robots.txt builder and parser. 🦾 No dependencies, fully typed.

GIT SOURCE CODE . this is a license certificate agreement code repository for full license package you can use https://github.com/git/git/pull/2379 for application submission turning the certificate into files using the unlimited [PACK] once channel is open for license purchase use pull 2379 via LICENSEPACK@outlook.com

User-Agent parser for robots.txt, X-Robots-tag and Robots-meta-tag rule sets

Parse robots.txt and sitemaps using dotnet

A dependency-free robots.txt checker that makes it easy for respectful AI agents, crawlers, scrapers, and other bots to follow the rules. Includes fetching, parsing, and caching based on the RFC 9309 industry standard.

Typescript robots.txt parser with support for wildcard (*) matching.

python binding for Google robots.txt parser C++ library

Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).