Home
x-crawl is a flexible Node.js AI-assisted crawler library. Flexible usage and powerful AI assistance functions make crawler work more efficient, intelligent and convenient.
It consists of two parts:
Crawler: It consists of a crawler API and various functions that can work normally even without relying on AI. AI: Currently based on the large AI model provided by OpenAI, AI simplifies many tedious operations.
If you find x-crawl helpful, or you like x-crawl, you can give x-crawl repository a like on GitHub A star. Your support is the driving force for our continuous improvement! thank you for your support!
- 🤖 AI Assistance - Powerful AI assistance function makes crawler work more efficient, intelligent and convenient.
- 🖋️ Flexible writing - A single crawling API is suitable for multiple configurations, and each configuration method has its own advantages.
- ⚙️Multiple uses - Supports crawling dynamic pages, static pages, interface data and file data.
- ⚒️ Control page - Crawling dynamic pages supports automated operations, keyboard input, event operations, etc.
- 👀 Device Fingerprinting - Zero configuration or custom configuration to avoid fingerprint recognition to identify and track us from different locations.
- 🔥 Asynchronous Sync - Asynchronous or synchronous crawling mode without switching crawling API.
- ⏱️ Interval crawling - no interval, fixed interval and random interval, determine whether to crawl with high concurrency.
- 🔄 Failed Retry - Customize the number of retries to avoid crawling failures due to temporary problems.
- ➡️ Rotation proxy - Automatic proxy rotation with failed retries, custom error times and HTTP status codes.
- 🚀 Priority Queue - Based on the priority of a single crawl target, it can be crawled ahead of other targets.
- 🧾 Crawl information - Controllable crawl information, which will output colored string information in the terminal.
- 🦾 TypeScript - Own types and implement complete types through generics.