A list of AI agents and robots to block.

Spiderbuf 是一个专注于 Python 爬虫练习的网站。提供丰富的爬虫教程、爬虫案例解析和爬虫练习题。Python爬虫开发强化练习，在矛与盾的攻防中不断提高技术水平，通过大量的爬虫实战掌握常见的爬虫与反爬套路。 引导式爬虫案例 + 免费爬虫视频教程，以闯关的形式挑战各个爬虫任务，培养爬虫开发的直觉及经验，验证自身爬虫开发与反爬虫实力的时候到了。

Proxy List Scrapper

Vietnamese text data crawler scripts for various sites (including Youtube, Facebook, 4rum, news, ...)

hproxy - Asynchronous IP proxy pool, aims to make getting proxy as convenient as possible.(异步爬虫代理池)

Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis

Python script to check if there is any differences in responses of an application when the request comes from a search engine's crawler.

对 ProxyPool 代理的二次魔改，在原支持HTTP代理基础上，增加了对SOCKS4、SOCKS5代理协议支持，增加了完善了fetch来源，改善了部分功能，如/count请求、server和schedule模式形成守护进程等，优化修改了部分bug。

API que busca dados de um processo em todos os graus dos Tribunais de Justiça de Alagoas (TJAL) e do Ceará (TJCE).

htcap is a web application scanner able to crawl single page application (SPA) in a recursive manner by intercepting ajax calls and DOM changes.

Licitações de Feira de Santana de fácil acesso aos cidadãos 🏦

Detect bots/crawlers/spiders via user-agent string

A web crawler which crawls the stackoverflow website.

Scraping the wiki pages and find the minimum number of links between two wiki pages

Small Simple Lightweight Spider Framework

🕷️ | dezcrwl is a website history crawler gather hidden information and check vulnerabilities for extracted .js endpoints & much more!

A Web Crawler developed in Python.

Web Crawler with Python

A bot to login in Twitter and process page with selenium using Python.

Article of WeChat to RSS