No timeout in urllib.robotparser with focused_crawler

[JER-CE]

Some URLs will cause the focused_crawler to never return anything. The below code runs indefinitely:

from trafilatura.spider import focused_crawler
import logging

logging.basicConfig(level=logging.DEBUG)
url = 'https://www.maersk.com/news/category/press-releases'
focused_crawler(url, max_seen_urls=1, max_known_urls=100)

[adbar]

Good point, it's not a bug in itself, the feature is not implemented yet. Let's put that on the list.

[adbar]

@JER-CE It was actually a bug, urllib.robotparser can try to load the URL indefinitely. Thanks for the detailed code snippet, it really helps.

Now it's still not working (on my computer at least) but we know why while debugging: the download fails (probably a user-agent issue).