Skip to content
#

scraping-tools

Star

Here is 1 public repository matching this topic...

Language: Shell
Filter by language
All 9 Python 3 JavaScript 2 Shell 1 TypeScript 1

Pryodon / Web-Spider-Linux-shell-script

Star 2

Generate a list of file links you can feed to wget for easy downloading! Mainly used for spidering web folders with lots of files. Can even generate a sitemap.txt or XML file for your website!

bash crawler scraper spider web-crawler web-scraper wget wget-utility scrapers bash-script webcrawler web-scrapers spiders webspider bash-scripting web-spider scraping-tool web-spidering webcrawlers scraping-tools
  • Updated Oct 9, 2025
  • Shell

Improve this page

Add a description, image, and links to the scraping-tools topic page so that developers can more easily learn about it.

Curate this topic

Add this topic to your repo

To associate your repository with the scraping-tools topic, visit your repo's landing page and select "manage topics."

Learn more