Clarify content types supported by SitemapRequestLoader

[loic-bellinger]

Quick question: does SitemapRequestLoader officially support only XML and plaintext sitemaps? From the code it selects text/plain or .txt for a line-based parser and defaults to XML otherwise. It seems text/html “sitemaps” (HTML pages listing links) aren’t parsed. Is that intended (I guess yes since the crawler would anyway probably properly discover and parse them), and could this be clarified in the docs ?

Current use case for context: i have a raw domains list that I cannot trust, so I have to test valid start seeds and also discovers the potential sitemaps (they may be missing from robots.txt and contain links that cannot be reached by classic crawling).

I then go that way

sitemap_loader = SitemapRequestLoader(
    http_client=http_client,
    sitemap_urls=sitemap_urls,
)
request_manager = await sitemap_loader.to_tandem()
await request_manager.add_requests(seeds_urls)

[janbuchar]

Hello @loic-bellinger. The SitemapRequestLoader is intended to handle sitemaps that follow the format described in https://www.sitemaps.org/protocol.html. Sitemaps in the "huge HTML full of links" fall in the scope of regular crawlers and enqueue_links, as you correctly assumed.

I believe this is something we should make clear in the docs - thank you for the input 🙂