Scrapy is a open source python package, it can build a spider to crawl all the websites thay you're interested.
Some tutorials are beblow:
https://city.shaform.com/zh/2017/05/13/scrapy-cloud/
https://chenqx.github.io/2014/12/23/Spider-Advanced-for-Dynamic-Website-Crawling/
https://doc.scrapy.org/en/latest/intro/overview.html#topics-whatelse