A Python library to utilize AWS API Gateway's large IP pool as a proxy to generate pseudo-infinite IPs for web scraping and brute forcing.

newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:

fast python port of arc90's readability tool, updated to match latest readability.js!

Extraction of the journalistic five W and one H questions (5W1H) from news articles: who did what, when, where, why, and how?

news-please - an integrated web crawler and information extractor for news that just works

Lightweight scraper for Google News

Twitter's API made free using puppeteer

A social networking service scraper in Python

2025! X / Twitter API scrapper with authorization support. Allows you to scrape search results, User's profiles (followers/following), Tweets (favoriters/retweeters) and more.

Twitter Scraper

Implementation of X/Twitter v1, v2, and GraphQL APIs

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

Playwright Model Context Protocol Server - Tool to automate Browsers and APIs in Claude Desktop, Cline, Cursor IDE and More 🔌