Sitemap Scraper issues

[nicklamonov]

Multiple problems with Sitemap Scraper have been discovered during testing:

1. Stuck runs (reason unknown)
a. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/5wHZZusa6D3hgQbYV#log (https://warehouse-theme-metal.myshopify.com/ - sitemap is there, but maybe format of urls is not being processed)
b. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/hc53VPklsgZN2Qk3B#log (https://seomator.com/ - sitemap looks fine)
c. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/XMD4gDBACklbC2rvV#log (https://sitegpt.ai/ - sitemap look ok. it's a sitemap index, but again pretty valid)
d. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/v2iLnieKCh3BqDKsY#log (https://landing-page.io/ - sitemap look ok. it's a sitemap index, but again pretty valid)
e. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/KUDhmzm5yGlAx1oI6#log (https://www.screamingfrog.co.uk/ - sitemap looks ok)
f. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/mdOWlrNXTY4pB4sT4#log (https://www.hubspot.com/ - sitemap looks ok)

2. No valid sitemaps discovered
a. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/u7cPO18EoHAmMKbfg#log (https://www.alza.cz/ - there are a bunch of sitemaps in robots.txt, which weren't discovered)
a. 1. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/3xuZgcqm6xwaxWOpb#log (when I give it a direct URL to a sitemap, it also doesn't process it)
b. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/wb8pznVQB0KZC0ivI#log (https://www.datart.cz/ - there is a sitemap in robots.txt, which wasn't discovered)
b.1. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/ATFC1iAhTdomwsb3C#input (when I give it a direct URL to a sitemap, it also doesn't process it)
c. https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/RPwTeiK0h9UFBiH7d#log (https://backlinko.com/ - sitemap is mentioned in robots.txt, but is not picked up)

3. Large sitemap throws bunch of errors
https://console.apify.com/organization/ZscMwFR5H7eCtWtyh/actors/rGeTNESChDZ65EbYh/runs/L3xe0rTA2iYS7NcYN#log (https://www.nytimes.com/ - it picks up sitemaps ok, start processing, but after some time starts to throw many warnings/errors related to performance limits. also I'm not sure if it processes .xml.gz sitemaps correctly. this run was too large to evaluate it)

[foxt451]

regarding cluster 1. - they block gotScraping, but not got/fetch. Since scraping sitemaps is expected to be done by bots, I'd switch to plain got, but would first look into other cases

[foxt451]

2. a) the site shows cloudflare challenge for requests both with regular got/fetch clients and with got-scraping, and even when manually going there with a browser (for me). maybe they only allow for some known bots
   b) the site shows a challenge for robots.txt
   b1) I forgot to extend the regex list for checking if the provided url is a sitemap with sitemap_index.xml, but it's apparently a less common name here (sitemapindex.xml). I'd consider just plainly considering all sitemap*.xml names a sitemap
   c) problem similar to cluster 1 for sitemap files

[foxt451]

re. 3 I'll look into it

but for the cluster 2 I'd suggest to provide no solution until crawlee v4 is released and we can pass in impit http client to bypass detection, and for 1 I'd suggest to switch to regular got for now

[nicklamonov]

@barjin , what do you think about the issues and proposed solutions/workarounds?

[barjin]

Since scraping sitemaps is expected to be done by bots, I'd switch to plain got

100% agree, let's make the stack as slim as possible 👍 I'd even go with vanilla fetch (or impit, if you need proxies and native Crawlee integration). Note that @crawlee/impit-client is a package you can install now and use with Crawlee v3 - see the linked docs for more info.

shows a challenge for robots.txt

I would just ignore these then, the file's literally called robots.txt, if they don't want the robots to scrape them, too bad :)

checking if the provided URL is a sitemap

In WCC, when somebody submits a sitemap as a start URL, we load it with a regular request and check if the content-type header was XML and if the file contains the right tags (urlset and whatnot). The regex approach can cause issues with URLs like https://en.wikipedia.org/wiki/sitemap.xml (a regular HTML page).

[barjin]

Related to apify/crawlee#3389 (comment)

Since this is small enough, new Actor, can we try dogfooding Crawlee v4 here? 👀 v4 has support for alternative HttpClient instances in the Sitemap-related utils

cc @nikitachapovskii-dev

[nicklamonov]

@barjin , can we already use some current version of v4 (beta / alpha or anything else available)?
We're already behind schedule to release this Actor, so if there is anything unstable in v4 which needs more time to finish before we can reference it in this Actor, then I'd consider other options.
But if we're going to use just sitemap-related capabilities, then it could be possibly safe and justified.