A python library to recursively crawl every sitemap.xml for a website. Also handles robots.txt and other well-knowns.
-
Updated
Oct 23, 2024 - Python
#
well-known
Here are 2 public repositories matching this topic...
Generate well-known nodeinfo files for an ATProtocol PDS.
-
Updated
Sep 11, 2024 - Python
Improve this page
Add a description, image, and links to the well-known topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the well-known topic, visit your repo's landing page and select "manage topics."