README.md

Web-scraping-price

Running the program:

python3 scraper.py

Dependencies:

• amazon.py
• flipkart.py
• newegg.py

Outputs:

• amazon-laptops.csv
• flipkart-laptops.csv
• newegg-laptops.csv

ToDo:

• Factorise the code to use in multiple sites and urls

• Find more sites to scrap the data

• Create scraper functions for all these sites

• Use Pandas to manipulate the data

• Extract useful information from the data

• Automate the script to run periodically

• make ratings a comparable float

• fix when no discount, current price is differently formed

• Use title to extract their specs

• Sorting the data accordingly to select best option available

• Write script for Flipkart

• Find other good sites available to buy new/old laptops

• put urlopen in try/except or while loop

• add link to the product in csv

• use next page as a url too, to extract information

• Report the best options from sorted data

• Automate to run everyday/periodically

• Add Refurbished and renewed labels

• get the last page number from the page and create list of urls to scrape

• stop newegg after certain page number

• stop flipkart after certain page number

• robot check problem