Massively parallel serverless web scraper with adaptive WAF evasion.
Uses AWS Lambda for IP diversity — each invocation gets a fresh datacenter IP with full browser TLS fingerprint impersonation via curl_cffi. An async orchestrator manages hundreds of concurrent Lambda workers with real-time WAF rate monitoring and adaptive throttling.
Local Machine AWS Lambda (N instances)
┌──────────────────┐ ┌─────────────────────┐
│ Orchestrator │ ──── invoke ──► │ handler.py │
│ (async Python) │ │ curl_cffi + zstd │
│ │ ◄── response ── │ 15 browser profiles│
│ - Priority queue│ │ WAF detection │
│ - WAF throttle │ │ Fresh IP per call │
│ - Multi-region │ └─────────────────────┘
│ - Stats tracking│
│ - Local parsing │
└──────────────────┘
- 15 browser fingerprint profiles with coherent TLS + User-Agent + sec-ch-ua headers
- Adaptive WAF throttling — auto-jitter at 5 blocks/min, pause at 15/min
- Per-retry exponential backoff — 0.5s → 1s → 2s → 4s → 8s
- Multi-region support — distribute workers across AWS regions for IP diversity
- Graceful shutdown — Ctrl+C finishes in-flight requests and saves report
- Pluggable parsers — regex-based data extraction with greedy universal parser
- Pre-filtered ID lists — CSV-based target generation eliminates 35-40% wasted requests
- HTML caching — gzip-compressed local cache for offline re-parsing
- Per-fingerprint WAF tracking — identify which browser profiles get blocked most
| Metric | Tor (v3) | Lambda Swarm (v4) |
|---|---|---|
| Throughput | 315K-600K pages/hr | 800K-1.2M pages/hr |
| WAF block rate | 60-70% | <1% |
| IP diversity | 30 Tor exits | Unlimited (fresh per call) |
| Setup | 30 local Tor instances | deploy.sh once |
cd lambda_function/
./deploy.sh --create # First time
./deploy.sh # Updates# Point to your CSV data directory
export LAMBDA_SWARM_CSV_DIR=/path/to/csv/data
# Generate ID lists for all targets
lambda-swarm generate-ids
# Or specific targets
lambda-swarm generate-ids --targets npc,spell,item# Smoke test (50 pages)
lambda-swarm scrape --targets npc --smoke 50
# Full run with 500 concurrent workers
lambda-swarm scrape --targets npc,quest,item --concurrency 500
# Multi-region for maximum IP diversity
lambda-swarm scrape --targets npc --regions us-east-1,us-west-2,eu-west-1lambda-swarm scrape --targets npc --reparsepip install -e .Or just install dependencies:
pip install -r requirements.txt| Environment Variable | Default | Description |
|---|---|---|
AWS_REGION |
us-east-1 |
Default AWS region |
LAMBDA_FUNCTION_NAME |
lambda-swarm |
Lambda function name |
LAMBDA_SWARM_CSV_DIR |
./csv_data |
CSV data directory for ID generation |
LAMBDA_SWARM_BUILD_ID |
latest |
Build identifier |
- Add URL pattern to
lambda_swarm/config.py→TARGET_CONFIGS - Add a parser to
lambda_swarm/parsers.py(or useparse_generic_page) - Add CSV mapping to
lambda_swarm/config.py→TARGET_DB2_MAP - Generate ID list:
lambda-swarm generate-ids --targets your_target
lambda-swarm/
├── lambda_swarm/ # Python package
│ ├── orchestrator.py # Async engine (multi-region, adaptive WAF)
│ ├── parsers.py # Pluggable page parsers (18 specialized + generic)
│ ├── config.py # Target configs, URL patterns, DB2 mappings
│ ├── csv_loader.py # CSV data loading for ID generation
│ ├── generate_ids.py # ID list generation with density analysis
│ └── cli.py # CLI entry point
├── lambda_function/ # AWS Lambda worker
│ ├── handler.py # Fetch worker (curl_cffi + zstd)
│ ├── Dockerfile # Container definition
│ ├── deploy.sh # ECR build + Lambda deploy
│ └── requirements.txt # Lambda-side dependencies
├── id_lists/ # Generated ID lists (gitignored)
├── output/ # Scraped data (gitignored)
└── reports/ # Scrape summary reports
MIT