CNN Web Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# INSY 5336 Python Programming Spring 2021 Final Term Project "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Algorithm:\n",
    "1. import urllib.request, urllib.parse, urllib.error for opening urls to retrieve data, BeautifulSoup for web scrapping by parsing html/xml, ssl for secure communication, csv for reading and writing to csv files.\n",
    "2. Create a default context object for secure communication.\n",
    "3. Define get_stock_data_and_save() function which takes category name and category's dictionary as input. In the function: \n",
    "    i) Open a stocks.csv in append mode as we would have already written the headers in the file in the main() function.\n",
    "    ii) Iterate through the category's dictionary to get the quote for each ticker symbol from its detail page using BeautifulSoup and save the details along with stock name in stocks.csv\n",
    "    iii) close the csv file.\n",
    "4. Define main() function.In the function: \n",
    "    i) Open the https://money.cnn.com/data/hotstocks/ url using urllib.request.urlopen()\n",
    "    ii) Create the BeautifulSoup object\n",
    "    iii) Get the list of Most Actives, Gainers and Losers from https://money.cnn.com/data/hotstocks/ website using the soup object.\n",
    "    iv) Write the headers to the csv file in the first row\n",
    "    v) Call get_stock_data_and_save() function for Most Actives, Gainers and Losers.\n",
    "    vi) Print the list of Most Actives, Gainers and Losers.\n",
    "    vii) Get data from stocks.csv and save it in a dictionary.\n",
    "    viii) Ask the user to enter the ticker symbol to print its details. If user does not enter a valid ticker, print error and ask him to enter it again.\n",
    "    ix)  If user enters correct ticker, print the details.\n",
    "5. Call main().\n",
    "\n",
    "Instructions to run the code: (The program takes a while to run. Please be patient)\n",
    "1. Execute the cell below.\n",
    "2. First the program will scrape the https://money.cnn.com/data/hotstocks/ website to get a list of Most Actives, Gainers and Losers.\n",
    "3. Then for each stock in Most Actives, Gainers and Losers, stock's detail page will be scraped to get open, close, volume, market cap and this data will be saved in stocks.csv\n",
    "4. Then user will be asked to input a ticker symbol to print its details.If user does not enter a valid ticker, error will be printed and he will be asked to enter it again. If user enters correct ticker, details of that stock will be printed.\n",
    "\n",
    "Program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Which stock are you interested in:\n",
      "\n",
      "Most Actives:\n",
      "GE General Electric Co\n",
      "F Ford Motor Co\n",
      "BAC Bank of America Corp\n",
      "PFE Pfizer Inc\n",
      "T AT&T Inc\n",
      "CCL Carnival Corp\n",
      "TWTR Twitter Inc\n",
      "FCX Freeport-McMoRan Inc\n",
      "WFC Wells Fargo & Co\n",
      "XOM Exxon Mobil Corp\n",
      "\n",
      "\n",
      "Gainers:\n",
      "IT Gartner Inc\n",
      "SEE Sealed Air Corp\n",
      "LEG Leggett & Platt Inc\n",
      "BEN Franklin Resources Inc\n",
      "AMCR Amcor PLC\n",
      "MLM Martin Marietta Materials Inc\n",
      "VMC Vulcan Materials Co\n",
      "NUE Nucor Corp\n",
      "EL Estee Lauder Companies Inc\n",
      "CVS CVS Health Corp\n",
      "\n",
      "\n",
      "Losers:\n",
      "CTLT Catalent Inc\n",
      "PAYC Paycom Software Inc\n",
      "CCL Carnival Corp\n",
      "RCL Royal Caribbean Cruises Ltd\n",
      "DAL Delta Air Lines Inc\n",
      "KR Kroger Co\n",
      "GNRC Generac Holdings Inc\n",
      "LYV Live Nation Entertainment Inc\n",
      "GPN Global Payments Inc\n",
      "TYL Tyler Technologies Inc\n",
      "\n",
      "\n",
      "User inputs: twt\n",
      "Please enter a valid stock ticker!\n",
      "User inputs: twtr\n",
      "The data for TWTR Twitter Inc is the following:\n",
      "TWTR Twitter Inc\n",
      "OPEN:  55.07\n",
      "PREV CLOSE:  54.58\n",
      "VOLUME:  30,404,484\n",
      "MARKET CAP:  $44.1B\n"
     ]
    }
   ],
   "source": [
    "# for opening urls to retrieve data\n",
    "import urllib.request, urllib.parse, urllib.error\n",
    "# for web scrapping by parsing html/xml\n",
    "from bs4 import BeautifulSoup\n",
    "# to use OpenSSL for secure communication\n",
    "import ssl\n",
    "# for reading and writing to csv files\n",
    "import csv\n",
    "\n",
    "# create default context for secure communication\n",
    "ctx = ssl.create_default_context()\n",
    "ctx.check_hostname = False\n",
    "ctx.verify_mode = ssl.CERT_NONE\n",
    "  \n",
    "# this function take a dictionary as input and gets the details of each ticker symbol in the dictionary using its url and saves the details in stocks.csv\n",
    "def get_stock_data_and_save(category,dict_name):\n",
    "    # open csv file named stocks.csv in append mode as the headers are already written to the csv\n",
    "    csv_file = open('stocks.csv', 'a', newline = '')\n",
    "    csv_writer = csv.writer(csv_file)\n",
    "    # get the quote for each ticker symbol in the input dictionary and save the details along with stock name in stocks.csv\n",
    "    # one row per ticker symbol in the csv\n",
    "    for sym,value in dict_name.items():\n",
    "        # url to get the quote for the ticker symbol\n",
    "        detail_page_url = 'https://money.cnn.com'+value[1]\n",
    "        # open the url for scrapping quote for the ticker symbol\n",
    "        detail_page_html = urllib.request.urlopen(detail_page_url, context=ctx).read() \n",
    "        # create the BeautifulSoup object\n",
    "        detail_page_soup = BeautifulSoup(detail_page_html, 'lxml')\n",
    "        # find h3 whose text is 'Today’s Trading'\n",
    "        detail_page_header = detail_page_soup.find('h3', text = 'Today’s Trading')\n",
    "        # use find_next_sibling() method to get the table for the quote as its tag is parallel to h3 tag\n",
    "        detail_page_table = detail_page_header.find_next_sibling('div').find('table')\n",
    "        # get open price\n",
    "        todays_open = detail_page_table.find('td', text = 'Today’s open').find_next_sibling('td').text\n",
    "        # get previous close\n",
    "        prev_close = detail_page_table.find('td', text = 'Previous close').find_next_sibling('td').text\n",
    "        # get volume\n",
    "        volume = detail_page_table.find('td', text = 'Volume').find_next_sibling('td').text\n",
    "        # get market cap\n",
    "        market_cap = detail_page_table.find('td', text = 'Market cap').find_next_sibling('td').text\n",
    "        # write the details to csv in a row for this ticker symbol\n",
    "        csv_writer.writerow([category,sym, value[0], todays_open, prev_close, volume, market_cap])\n",
    "    #close the csv file\n",
    "    csv_file.close()\n",
    "      \n",
    "# this function gets the list of Most Actives, Gainers and Losers from https://money.cnn.com/data/hotstocks/ website and prints it\n",
    "# it then calls get_stock_data_and_save() function to save the details for each stock\n",
    "# it asks the user to input a ticker symbol and if user enters correct ticker, print the details\n",
    "def main():\n",
    "    # open the url for scrapping Most Actives, Gainers and Losers\n",
    "    url = 'https://money.cnn.com/data/hotstocks/'\n",
    "    html = urllib.request.urlopen(url, context=ctx).read() \n",
    "    # create the BeautifulSoup object\n",
    "    soup = BeautifulSoup(html, 'lxml')\n",
    "\n",
    "    # Code to scrape Most Actives\n",
    "    # find h3 whose text is 'Most Actives'\n",
    "    header = soup.find('h3', text = 'Most Actives')\n",
    "    # use find_next_sibling() method to get the table for Most Actives as its tag is parallel to the above h3 tag\n",
    "    table = header.find_next_sibling('table')\n",
    "    # define a list to store the Most Actives stock names with the ticker symbol\n",
    "    MostActives_list = []\n",
    "    # define a dictionary with key as Most Actives stock's ticker symbol and value as a tuple of stock's title and detailed page url\n",
    "    MostActives_dict = {}\n",
    "    # iterate through all the trs for this table to get the Most Actives stocks\n",
    "    table_rows = table.find_all('tr')\n",
    "    for tr in table_rows:\n",
    "        if tr.find('td') != None:\n",
    "            td = tr.find('td')\n",
    "            # append stock names with the ticker symbol\n",
    "            MostActives_list.append(td.text)\n",
    "            # get the ticker symbol\n",
    "            ticker_symbol = td.a.text\n",
    "            # get the url of the detail page\n",
    "            ticker_link = td.a.get('href')\n",
    "            # get the stock title\n",
    "            ticker_title = td.span.get('title')\n",
    "            # add to the dictionary with key as Most Actives stock's ticker symbol and value as a tuple of stock's title and detail page url\n",
    "            MostActives_dict[ticker_symbol] = (ticker_title,ticker_link)\n",
    "            \n",
    "    # Code to scrape Gainers\n",
    "    # find h3 whose text is 'Gainers'\n",
    "    header = soup.find('h3', text = 'Gainers')\n",
    "    # use find_next_sibling() method to get the table for Gainers as its tag is parallel to the above h3 tag\n",
    "    table = header.find_next_sibling('table')\n",
    "    # define a list to store the Gainers stock names with the ticker symbol\n",
    "    Gainers_list = []\n",
    "    # define a dictionary with key as Gainers stock's ticker symbol and value as a tuple of stock's title and detailed page url\n",
    "    Gainers_dict = {}\n",
    "    # iterate through all the trs for this table to get the Gainers stocks\n",
    "    table_rows = table.find_all('tr')\n",
    "    for tr in table_rows:\n",
    "        if tr.find('td') != None:\n",
    "            td = tr.find('td')\n",
    "            # append stock names with the ticker symbol\n",
    "            Gainers_list.append(td.text)\n",
    "            # get the ticker symbol\n",
    "            ticker_symbol = td.a.text\n",
    "            # get the url of the detail page\n",
    "            ticker_link = td.a.get('href')\n",
    "            # get the stock title\n",
    "            ticker_title = td.span.get('title')\n",
    "            # add to the dictionary with key as Gainers stock's ticker symbol and value as a tuple of stock's title and detail page url\n",
    "            Gainers_dict[ticker_symbol] = (ticker_title,ticker_link)\n",
    "            \n",
    "    # Code to scrape Losers\n",
    "    # find h3 whose text is 'Losers'\n",
    "    header = soup.find('h3', text = 'Losers')\n",
    "    # use find_next_sibling() method to get the table for Losers as its tag is parallel to the above h3 tag\n",
    "    table = header.find_next_sibling('table')\n",
    "    # define a list to store the Losers stock names with the ticker symbol\n",
    "    Losers_list = []\n",
    "    # define a dictionary with key as Losers stock's ticker symbol and value as a tuple of stock's title and detailed page url\n",
    "    Losers_dict = {}\n",
    "    # iterate through all the trs for this table to get the Losers stocks\n",
    "    table_rows = table.find_all('tr')\n",
    "    for tr in table_rows:\n",
    "        if tr.find('td') != None:\n",
    "            td = tr.find('td')\n",
    "            # append stock names with the ticker symbol\n",
    "            Losers_list.append(td.text)\n",
    "            # get the ticker symbol\n",
    "            ticker_symbol = td.a.text\n",
    "            # get the url of the detail page\n",
    "            ticker_link = td.a.get('href')\n",
    "            # get the stock title\n",
    "            ticker_title = td.span.get('title')\n",
    "            # add to the dictionary with key as Losers stock's ticker symbol and value as a tuple of stock's title and detail page url\n",
    "            Losers_dict[ticker_symbol] = (ticker_title,ticker_link)\n",
    "         \n",
    "    # open csv file named stocks.csv in write mode\n",
    "    csv_file = open('stocks.csv', 'w', newline = '')\n",
    "    csv_writer = csv.writer(csv_file)\n",
    "    # write the headers to the csv file in the first row\n",
    "    csv_writer.writerow(['Category','Ticker Symbol', 'Stock Name', 'Open', 'Prev Close', 'Volume', 'Market Cap'])\n",
    "    #close the csv file\n",
    "    csv_file.close()    \n",
    "    # call get_stock_data_and_save() function to get details for each Most Actives stock, Gainers stock and Losers stock and save it in csv\n",
    "    get_stock_data_and_save(\"Most Actives\",MostActives_dict)\n",
    "    get_stock_data_and_save(\"Gainers\",Gainers_dict)\n",
    "    get_stock_data_and_save(\"Losers\",Losers_dict)\n",
    "    # ask the user for the ticker symbol to view the details\n",
    "    print('Which stock are you interested in:\\n')\n",
    "    # print the Most Actives stocks with their ticker symbol\n",
    "    print('Most Actives:')\n",
    "    for stock in MostActives_list:\n",
    "        print(stock)\n",
    "    print('\\n')\n",
    "    # print the Gainers stocks with their ticker symbol\n",
    "    print('Gainers:')\n",
    "    for stock in Gainers_list:\n",
    "        print(stock)\n",
    "    print('\\n')\n",
    "    # print the Losers stocks with their ticker symbol\n",
    "    print('Losers:')\n",
    "    for stock in Losers_list:\n",
    "        print(stock)\n",
    "    print('\\n')\n",
    "    \n",
    "    # open stocks.csv to get all the data from the csv file into a dictionary with key as ticker symbol and value as its detail\n",
    "    csv_file = open('stocks.csv')\n",
    "    csv_reader = csv.reader(csv_file, delimiter=',')\n",
    "    csv_data = {}\n",
    "    for line in csv_reader:\n",
    "        csv_data[line[1]] = {}\n",
    "        csv_data[line[1]]['title'] = line[1]+\" \"+line[2]\n",
    "        csv_data[line[1]]['open'] = line[3]\n",
    "        csv_data[line[1]]['prev_close'] = line[4]\n",
    "        csv_data[line[1]]['volume'] = line[5]\n",
    "        csv_data[line[1]]['market_cap'] = line[6]\n",
    "    # close the csv\n",
    "    csv_file.close()\n",
    "    # if user does not enter a valid ticker, print error and ask him to enter it again\n",
    "    while True:\n",
    "        user_input = input('User inputs: ')\n",
    "        # remove whitespaces from user_input\n",
    "        user_input = user_input.upper().strip()\n",
    "        if user_input not in csv_data:\n",
    "            print(\"Please enter a valid stock ticker!\")\n",
    "        else:\n",
    "            # if user enters correct ticker, print the details\n",
    "            print(\"The data for\",csv_data[user_input]['title'],\"is the following:\")\n",
    "            print(csv_data[user_input]['title'])\n",
    "            print('OPEN: ',csv_data[user_input]['open'])\n",
    "            print('PREV CLOSE: ',csv_data[user_input]['prev_close'])\n",
    "            print('VOLUME: ',csv_data[user_input]['volume'])\n",
    "            print('MARKET CAP: ',csv_data[user_input]['market_cap'])\n",
    "            break\n",
    "\n",
    "# call the main function to begin execution\n",
    "main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}