webcrawler A rudimentary web crawler This repository contains a distributed web crawling system for systematic website analysis.