A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).
Web Application Security Scanner Framework
Kimurai is a modern web scraping framework written in Ruby which works out of box with Headless Chromium/Firefox, PhantomJS, or simple HTTP requests and allows to scrape and interact with JavaScript rendered websites
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Tanakai is a modern web scraping framework written in Ruby. A fork of Kimurai.
Crawl instagram photos, posts and videos for download.
Ruby gem to detect bots and crawlers via the user agent
⏬ Yet another torrent searching application for your command line
Competitive programming contests schedule
Gera o arquivo CSV para importar / migrar do filmow para o Letterboxd
your friendly neighborhood web crawler
- THIS IS AN OLD FORK - Checkout Medusa Crawler gem instead "medusa-crawler"
Ruby Email Extractor - a small ruby class to grab some emails from a URL by screen scraping.
Fast Crawlbase API crawling library