Scrapy, a fast high-level web crawling & scraping framework for Python.

Python ProxyPool for web spider

A Powerful Spider(Web Crawler) System in Python.

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。

Incredibly fast crawler designed for OSINT.

Python脚本。模拟登录知乎， 爬虫，操作excel，微信公众号，远程开机

Use LLMs to dig out what you care about from massive amounts of information and a variety of sources daily.

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

基于搜狗微信搜索的微信公众号爬虫接口

Redis-based components for Scrapy.

Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.

💖 High available distributed ip proxy pool, powerd by Scrapy and Redis

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼、阿里任务、博客园、微博、百度贴吧、豆瓣电影、包图网、全景网、豆瓣音乐、某省药监局、搜狐新闻、机器学习文本采集、fofa资产采集、汽车之家、国家统计局、百度关键词收录数、蜘蛛泛目录、今日头条、豆瓣影评、携程、小米应用商店、安居客、途家民宿❤️❤️❤️。微信爬虫展示项目:

A community-driven way to read and chat with AI bots - powered by chatGPT.

Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML

Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

Every web site provides APIs.