website-crawler

Broken link checker that crawls websites and validates links. Find broken links, dead links, and invalid URLs in websites, documentation, and local files. Perfect for SEO audits and CI/CD.

It allows you to download a website from the Internet to a local directory, building recursively all directories, getting HTML, images, and other files from the server to your computer.

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

Free SEO crawler & website audit tool — a self-hosted, open-source SEO spider that crawls any website for technical SEO issues. Concurrent, CMS-aware (Shopify/WordPress/Webflow), XLSX export. A free alternative to Screaming Frog.

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

Scraper for https://marvelsnapzone.com to retrieve metadata of Marvel SNAP cards.

A universal and local phishing toolkit for audit purposes

WebKnoGraph is an open research project that uses data processing, vector embeddings, and graph algorithms to optimize internal linking at scale. Built for both academic and industry use, it offers THE FIRST FULLY transparent, AI-driven framework for improving SEO and site navigation through reproducible methods.

A powerful Bash script for extracting URLs and API endpoints from HTML, JavaScript, and JSON content of web pages. Designed for security researchers, bug bounty hunters, and developers to streamline endpoint discovery. Simple to use, supports single or multiple URLs, and offers file-saving capabilities.

An almost generic web crawler built using Scrapy and Python 3.7 to recursively crawl entire websites.

Open-source website optimizer for SEO, AEO, GEO, performance, accessibility & technical audits — with AI-powered fixes, project auditing, crawling and 200+ rules that actually fixes the issues it finds

A Simple Script To Scrape DuckDuckGo Search Results Using Python And Selenium WebDriver.

A tutorial and code samples of web scraping with PHP

🕷️ | ReconX is a Live-Website Crawler made to gather critical information with an option to take a picture of each site crawled!

Native macOS and Windows workspace for technical SEO audits, site crawling, semantic maps, keyword research, rank tracking, DataForSEO, Search Console, and AI visibility.

Free, open-source desktop SEO crawler and technical site audit engine for macOS, Windows, and Linux. 250+ technical SEO, GEO, and AI-search checks with no page cap.

💫 Crawl urls from a webpage and provide a DomCrawler with Scraper Library

Web Link Crawler: A Python script to crawl websites and collect links based on a regex pattern. Efficient and customizable.

This a project to demonstrate the use of standard python libraries like os, urllib, HTMLParser to create a minimalist webpage crawler that crawls webpages on a website to gather hyperlinks (URLs)

Crawls a website to generate insights