Extracting Stock Data Using Web Scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/Logos/organization_logo/organization_logo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Extracting Stock Data Using Web Scraping</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all stock data is available via API in this assignment; you will use web-scraping to obtain financial data. You will be quizzed on your results.\\\n",
    "Using beautiful soup we will extract historical share data from a web-page.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "    <ul>\n",
    "        <li>Downloading the Webpage Using Requests Library</li>\n",
    "        <li>Parsing Webpage HTML Using BeautifulSoup</li>\n",
    "        <li>Extracting Data and Building DataFrame</li>\n",
    "    </ul>\n",
    "<p>\n",
    "    Estimated Time Needed: <strong>30 min</strong></p>\n",
    "</div>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: bs4 in c:\\users\\wawdh\\appdata\\local\\programs\\python\\python38\\lib\\site-packages (0.0.1)\n",
      "Requirement already satisfied: beautifulsoup4 in c:\\users\\wawdh\\appdata\\local\\programs\\python\\python38\\lib\\site-packages (from bs4) (4.9.3)\n",
      "Requirement already satisfied: soupsieve>1.2 in c:\\users\\wawdh\\appdata\\local\\programs\\python\\python38\\lib\\site-packages (from beautifulsoup4->bs4) (2.2.1)\n"
     ]
    }
   ],
   "source": [
    "#!pip install pandas\n",
    "#!pip install requests\n",
    "!pip install bs4\n",
    "#!pip install plotly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Example\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we must use the `request` library to downlaod the webpage, and extract the text. We will extract Netflix stock data <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html\"\n",
    "\n",
    "data  = requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we must parse the text into html using `beautiful_soup`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(data, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can turn the html table into a pandas dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "netflix_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "# First we isolate the body of the table which contains all the information\n",
    "# Then we loop through each row and find all the column values for each row\n",
    "for row in soup.find(\"tbody\").find_all('tr'):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    # Finally we append the data of each row to the table\n",
    "    netflix_data = netflix_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now print out the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78,560,600</td>\n",
       "      <td>528.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66,927,600</td>\n",
       "      <td>502.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111,573,300</td>\n",
       "      <td>513.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90,183,900</td>\n",
       "      <td>521.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61,902,300</td>\n",
       "      <td>538.85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low   Close       Volume Adj Close\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21   78,560,600    528.21\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81   66,927,600    502.81\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47  111,573,300    513.47\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66   90,183,900    521.66\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85   61,902,300    538.85"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the pandas `read_html` function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_html_pandas_data = pd.read_html(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beacause there is only one table on the page, we just take the first table in the list returned\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close*</th>\n",
       "      <th>Adj Close**</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78560600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66927600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111573300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90183900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61902300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low  Close* Adj Close**     Volume\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21      528.21   78560600\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81      502.81   66927600\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47      513.47  111573300\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66      521.66   90183900\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85      538.85   61902300"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_dataframe = read_html_pandas_data[0]\n",
    "\n",
    "netflix_dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Exercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `requests` library to download the webpage <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html>. Save the text of the response as a variable named `html_data`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "html_data = requests.get(\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html\").text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse the html data using `beautiful_soup`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(html_data, 'html5lib')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 1</b> What is the content of the title attribute:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>Amazon.com, Inc. (AMZN) Stock Historical Prices &amp; Data - Yahoo Finance</title>\n"
     ]
    }
   ],
   "source": [
    "print(soup.title)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using beautiful soup extract the table with historical share prices and store it into a dataframe named `amazon_data`. The dataframe should have columns Date, Open, High, Low, Close, Adj Close, and Volume. Fill in each variable with the correct data from the list `col`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "for row in soup.find(\"tbody\").find_all(\"tr\"):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0]#ADD_CODE\n",
    "    Open = col[1]#ADD_CODE\n",
    "    high = col[2]#ADD_CODE\n",
    "    low = col[3]#ADD_CODE\n",
    "    close = col[4]#ADD_CODE\n",
    "    adj_close = col[5]#ADD_CODE\n",
    "    volume = col[6]#ADD_CODE\n",
    "    \n",
    "    amazon_data = amazon_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the first five rows of the `amazon_data` dataframe you created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "               Date          Open          High           Low         Close  \\\n",
      "0  [[Jan 01, 2021]]  [[3,270.00]]  [[3,363.89]]  [[3,086.00]]  [[3,206.20]]   \n",
      "1  [[Dec 01, 2020]]  [[3,188.50]]  [[3,350.65]]  [[3,072.82]]  [[3,256.93]]   \n",
      "2  [[Nov 01, 2020]]  [[3,061.74]]  [[3,366.80]]  [[2,950.12]]  [[3,168.04]]   \n",
      "3  [[Oct 01, 2020]]  [[3,208.00]]  [[3,496.24]]  [[3,019.00]]  [[3,036.15]]   \n",
      "4  [[Sep 01, 2020]]  [[3,489.58]]  [[3,552.25]]  [[2,871.00]]  [[3,148.73]]   \n",
      "\n",
      "            Volume     Adj Close  \n",
      "0   [[71,528,900]]  [[3,206.20]]  \n",
      "1   [[77,556,200]]  [[3,256.93]]  \n",
      "2   [[90,810,500]]  [[3,168.04]]  \n",
      "3  [[116,226,100]]  [[3,036.15]]  \n",
      "4  [[115,899,300]]  [[3,148.73]]  \n"
     ]
    }
   ],
   "source": [
    "print(amazon_data.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 2</b> What is the name of the columns of the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Empty DataFrame\n",
      "Columns: [Date, Open, High, Low, Close, Volume, Adj Close]\n",
      "Index: []\n"
     ]
    }
   ],
   "source": [
    "print(amazon_data[-1:0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 3</b> What is the `Open` of the last row of the amazon_data dataframe?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0     [[3,270.00]]\n",
      "1     [[3,188.50]]\n",
      "2     [[3,061.74]]\n",
      "3     [[3,208.00]]\n",
      "4     [[3,489.58]]\n",
      "          ...     \n",
      "56      [[663.92]]\n",
      "57      [[590.49]]\n",
      "58      [[556.29]]\n",
      "59      [[578.15]]\n",
      "60      [[656.29]]\n",
      "Name: Open, Length: 61, dtype: object\n"
     ]
    }
   ],
   "source": [
    "print(amazon_data[:].get(\"Open\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n",
    "\n",
    "Azim Hirjani\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By    | Change Description        |\n",
    "| ----------------- | ------- | ------------- | ------------------------- |\n",
    "| 2021-06-09       | 1.2     | Lakshmi Holla|Added URL in question 3 |\n",
    "| 2020-11-10        | 1.1     | Malika Singla | Deleted the Optional part |\n",
    "| 2020-08-27        | 1.0     | Malika Singla | Added lab to GitLab       |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n",
    "\n",
    "<p>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}