PowerShell module for reading robots.txt files
-
Updated
May 18, 2022 - C#
#
robots-exclusion-protocol
Here are 3 public repositories matching this topic...
Parse robots.txt and sitemaps using dotnet
-
Updated
Sep 2, 2023 - C#
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
robots-txt
robots
meta-tags
robots-parser
robots-exclusion-standard
robotstxt
robots-tag
robots-exclusion-protocol
x-robots-tag
-
Updated
May 2, 2024 - C#
Improve this page
Add a description, image, and links to the robots-exclusion-protocol topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-exclusion-protocol topic, visit your repo's landing page and select "manage topics."