Add Scraper Script for Crawling Multiple Websites

Takes in a list of URLs from a text file and processes
them

Signed-off-by: Antony Oduor <aowino@gmail.com>

Author: ooduor

README.md

@@ -20,6 +20,9 @@ pip install tldextract
 pip install html5
 pip install pandas
 pip install tqdm
+pip install grequests
+pip install validators
+pip install tld
 ```
 
 ### keywords
@@ -55,3 +58,9 @@ their frequency and sorted.
 ```
 ./search.py -h
 ```
+
+
+####  Linux / MacOS / Windows Gotchas!
+
+
+https://stackoverflow.com/questions/19425857/env-python-r-no-such-file-or-directory

requirements.txt

@@ -13,3 +13,6 @@ webencodings==0.5.1
 Whoosh==2.7.4
 pandas==0.21.1
 tqdm==4.19.5
+grequests=0.3.0
+validators=0.12.0
+tld==0.7.9