Anemone is a web spider framework that can spider a domain and collect useful information about the pages it visits. It is versatile, allowing you to write your own specialized spider tasks quickly and easily.
See anemone.rubyforge.org for more information.
-
Multi-threaded design for high performance
-
Tracks 301 HTTP redirects
-
Built-in BFS algorithm for determining page depth
-
Allows exclusion of URLs based on regular expressions
-
Choose the links to follow on each page with focus_crawl()
-
HTTPS support
-
Records response time for each page
-
CLI program can list all pages in a domain, calculate page depths, and more
-
Obey robots.txt
-
In-memory or persistent storage of pages during crawl, using TokyoCabinet, SQLite3, MongoDB, or Redis
See the scripts under the lib/anemone/cli directory for examples of several useful Anemone tasks.
-
nokogiri
-
robots
To test and develop this gem, additional requirements are:
-
rspec
-
fakeweb
-
tokyocabinet
-
kyotocabinet-ruby
-
mongo
-
redis
-
sqlite3
You will need to have KyotoCabinet, Tokyo Cabinet, MongoDB, and Redis installed on your system and running.