A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

Web crawler designed to efficiently retrieve unique href, script and form links from a web application.

Generate a list of file links you can feed to wget for easy downloading! Mainly used for spidering web folders with lots of files. Can even generate a sitemap text or XML file for your website!