Final Assignment_Webscraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "    <img src=\"https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/Logos/organization_logo/organization_logo.png\" width=\"300\" alt=\"cognitiveclass.ai logo\"  />\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Extracting Stock Data Using a Web Scraping</h1>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not all stock data is available via API in this assignment; you will use web-scraping to obtain financial data. You will be quizzed on your results.\\\n",
    "Using beautiful soup we will extract historical share data from a web-page.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>Table of Contents</h2>\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "    <ul>\n",
    "        <li>Downloading the Webpage Using Requests Library</li>\n",
    "        <li>Parsing Webpage HTML Using BeautifulSoup</li>\n",
    "        <li>Extracting Data and Building DataFrame</li>\n",
    "    </ul>\n",
    "<p>\n",
    "    Estimated Time Needed: <strong>30 min</strong></p>\n",
    "</div>\n",
    "\n",
    "<hr>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "                  __    __    __    __\n",
      "                 /  \\  /  \\  /  \\  /  \\\n",
      "                /    \\/    \\/    \\/    \\\n",
      "███████████████/  /██/  /██/  /██/  /████████████████████████\n",
      "              /  / \\   / \\   / \\   / \\  \\____\n",
      "             /  /   \\_/   \\_/   \\_/   \\    o \\__,\n",
      "            / _/                       \\_____/  `\n",
      "            |/\n",
      "        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗\n",
      "        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗\n",
      "        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║\n",
      "        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║\n",
      "        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║\n",
      "        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝\n",
      "\n",
      "        mamba (0.15.3) supported by @QuantStack\n",
      "\n",
      "        GitHub:  https://github.com/mamba-org/mamba\n",
      "        Twitter: https://twitter.com/QuantStack\n",
      "\n",
      "█████████████████████████████████████████████████████████████\n",
      "\n",
      "\n",
      "Looking for: ['bs4==4.10.0']\n",
      "\n",
      "pkgs/main/noarch         [<=>                 ] (00m:00s) \n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 616 KB / ?? (1.98 MB/s)\n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 616 KB / ?? (1.98 MB/s)\n",
      "pkgs/r/linux-64          [<=>                 ] (00m:00s) \n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 616 KB / ?? (1.98 MB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 616 KB / ?? (1.98 MB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [<=>                 ] (00m:00s) \n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 616 KB / ?? (1.98 MB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 616 KB / ?? (1.98 MB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [<=>                 ] (00m:00s) \n",
      "pkgs/main/noarch         [=>                ] (00m:00s) 616 KB / ?? (1.98 MB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [=>               ] (00m:00s) 44 KB / ?? (144.95 KB/s)\n",
      "pkgs/main/noarch         [<=>                 ] (00m:00s) Finalizing...\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [=>               ] (00m:00s) 44 KB / ?? (144.95 KB/s)\n",
      "pkgs/main/noarch         [<=>                 ] (00m:00s) Done\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [=>               ] (00m:00s) 44 KB / ?? (144.95 KB/s)\n",
      "pkgs/main/noarch         [====================] (00m:00s) Done\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [=>               ] (00m:00s) 44 KB / ?? (144.95 KB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [<=>              ] (00m:00s) 44 KB / ?? (144.95 KB/s)\n",
      "pkgs/r/linux-64          [=>                ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/linux-64          [<=>               ] (00m:00s) 448 KB / ?? (1.44 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/linux-64          [ <=>                ] (00m:00s) 1 MB / ?? (2.22 MB/s)\n",
      "pkgs/r/noarch            [=>                ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/linux-64          [ <=>                ] (00m:00s) 1 MB / ?? (2.22 MB/s)\n",
      "pkgs/r/noarch            [<=>               ] (00m:00s) 376 KB / ?? (1.21 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/linux-64          [ <=>                ] (00m:00s) 1 MB / ?? (2.22 MB/s)\n",
      "pkgs/r/noarch            [<=>               ] (00m:00s) 784 KB / ?? (1.68 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/linux-64          [ <=>                ] (00m:00s) Finalizing...\n",
      "pkgs/r/noarch            [<=>               ] (00m:00s) 784 KB / ?? (1.68 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/linux-64          [ <=>                ] (00m:00s) Done\n",
      "pkgs/r/noarch            [<=>               ] (00m:00s) 784 KB / ?? (1.68 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/linux-64          [====================] (00m:00s) Done\n",
      "pkgs/r/noarch            [<=>               ] (00m:00s) 784 KB / ?? (1.68 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/noarch            [ <=>              ] (00m:00s) 784 KB / ?? (1.68 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/noarch            [  <=>               ] (00m:00s) 1 MB / ?? (1.89 MB/s)\n",
      "pkgs/main/linux-64       [<=>               ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/noarch            [  <=>               ] (00m:00s) 1 MB / ?? (1.89 MB/s)\n",
      "pkgs/main/linux-64       [ <=>              ] (00m:00s) 616 KB / ?? (1.32 MB/s)\n",
      "pkgs/r/noarch            [  <=>               ] (00m:00s) 1 MB / ?? (1.89 MB/s)\n",
      "pkgs/main/linux-64       [  <=>               ] (00m:00s) 1 MB / ?? (2.17 MB/s)\n",
      "pkgs/r/noarch            [  <=>               ] (00m:00s) Finalizing...\n",
      "pkgs/main/linux-64       [  <=>               ] (00m:00s) 1 MB / ?? (2.17 MB/s)\n",
      "pkgs/r/noarch            [  <=>               ] (00m:00s) Done\n",
      "pkgs/main/linux-64       [  <=>               ] (00m:00s) 1 MB / ?? (2.17 MB/s)\n",
      "pkgs/r/noarch            [====================] (00m:00s) Done\n",
      "pkgs/main/linux-64       [  <=>               ] (00m:00s) 1 MB / ?? (2.17 MB/s)\n",
      "pkgs/main/linux-64       [   <=>              ] (00m:00s) 1 MB / ?? (2.17 MB/s)\n",
      "pkgs/main/linux-64       [   <=>              ] (00m:00s) 2 MB / ?? (2.48 MB/s)\n",
      "pkgs/main/linux-64       [    <=>             ] (00m:00s) 2 MB / ?? (2.48 MB/s)\n",
      "pkgs/main/linux-64       [    <=>             ] (00m:00s) 3 MB / ?? (2.77 MB/s)\n",
      "pkgs/main/linux-64       [     <=>            ] (00m:00s) 3 MB / ?? (2.77 MB/s)\n",
      "pkgs/main/linux-64       [     <=>            ] (00m:00s) 3 MB / ?? (3.03 MB/s)\n",
      "pkgs/main/linux-64       [      <=>           ] (00m:00s) 3 MB / ?? (3.03 MB/s)\n",
      "pkgs/main/linux-64       [      <=>           ] (00m:00s) 4 MB / ?? (3.21 MB/s)\n",
      "pkgs/main/linux-64       [       <=>          ] (00m:01s) 4 MB / ?? (3.21 MB/s)\n",
      "pkgs/main/linux-64       [       <=>          ] (00m:01s) 4 MB / ?? (3.03 MB/s)\n",
      "pkgs/main/linux-64       [       <=>          ] (00m:01s) Finalizing...\n",
      "pkgs/main/linux-64       [       <=>          ] (00m:01s) Done\n",
      "pkgs/main/linux-64       [====================] (00m:01s) Done\n",
      "\n",
      "Pinned packages:\n",
      "  - python 3.7.*\n",
      "\n",
      "\n",
      "Transaction\n",
      "\n",
      "  Prefix: /home/jupyterlab/conda/envs/python\n",
      "\n",
      "  Updating specs:\n",
      "\n",
      "   - bs4==4.10.0\n",
      "   - ca-certificates\n",
      "   - certifi\n",
      "   - openssl\n",
      "\n",
      "\n",
      "  Package             Version  Build           Channel                  Size\n",
      "──────────────────────────────────────────────────────────────────────────────\n",
      "  Install:\n",
      "──────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[32m  + bs4           \u001b[00m     4.10.0  hd3eb1b0_0      pkgs/main/noarch        10 KB\n",
      "\n",
      "  Change:\n",
      "──────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[31m  - certifi       \u001b[00m  2022.6.15  py37h89c1867_0  installed                    \n",
      "\u001b[32m  + certifi       \u001b[00m  2022.6.15  py37h06a4308_0  pkgs/main/linux-64     153 KB\n",
      "\n",
      "  Upgrade:\n",
      "──────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[31m  - openssl       \u001b[00m     1.1.1p  h166bdaf_0      installed                    \n",
      "\u001b[32m  + openssl       \u001b[00m     1.1.1q  h7f8727e_0      pkgs/main/linux-64       3 MB\n",
      "\n",
      "  Downgrade:\n",
      "──────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[31m  - beautifulsoup4\u001b[00m     4.11.1  pyha770c72_0    installed                    \n",
      "\u001b[32m  + beautifulsoup4\u001b[00m     4.10.0  pyh06a4308_0    pkgs/main/noarch        85 KB\n",
      "\n",
      "  Summary:\n",
      "\n",
      "  Install: 1 packages\n",
      "  Change: 1 packages\n",
      "  Upgrade: 1 packages\n",
      "  Downgrade: 1 packages\n",
      "\n",
      "  Total download: 3 MB\n",
      "\n",
      "──────────────────────────────────────────────────────────────────────────────\n",
      "\n",
      "Downloading  [>                                        ] (00m:00s)   39.24 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished bs4                                  (00m:00s)              10 KB     39 KB/s\n",
      "Downloading  [>                                        ] (00m:00s)   39.24 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [>                                        ] (00m:00s)   39.24 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [>                                        ] (00m:00s)   39.24 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=>                                       ] (00m:00s)  367.83 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished beautifulsoup4                       (00m:00s)              85 KB    328 KB/s\n",
      "Downloading  [=>                                       ] (00m:00s)  367.83 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=>                                       ] (00m:00s)  367.83 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished certifi                              (00m:00s)             153 KB    593 KB/s\n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [==========>                              ] (00m:00s)        1 / 4\n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [==========>                              ] (00m:00s)        1 / 4\n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        2 / 4\n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        2 / 4\n",
      "Downloading  [===>                                     ] (00m:00s)  959.56 KB/s\n",
      "Extracting   [==============================>          ] (00m:00s)        3 / 4\n",
      "Downloading  [=====================>                   ] (00m:00s)    4.81 MB/s\n",
      "Extracting   [==============================>          ] (00m:00s)        3 / 4\n",
      "Downloading  [=====================>                   ] (00m:00s)    4.81 MB/s\n",
      "Extracting   [==============================>          ] (00m:00s)        3 / 4\n",
      "Downloading  [=========================================] (00m:00s)    8.90 MB/s\n",
      "Extracting   [==============================>          ] (00m:00s)        3 / 4\n",
      "\u001b[2A\u001b[0KFinished openssl                              (00m:00s)               3 MB      8 MB/s\n",
      "Downloading  [=========================================] (00m:00s)    8.90 MB/s\n",
      "Extracting   [==============================>          ] (00m:00s)        3 / 4\n",
      "Downloading  [=========================================] (00m:00s)    8.90 MB/s\n",
      "Extracting   [==============================>          ] (00m:00s)        3 / 4\n",
      "Downloading  [=========================================] (00m:00s)    8.90 MB/s\n",
      "Extracting   [==============================>          ] (00m:00s)        3 / 4\n",
      "Downloading  [=========================================] (00m:00s)    8.90 MB/s\n",
      "Extracting   [=========================================] (00m:00s)        4 / 4\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "\n",
      "                  __    __    __    __\n",
      "                 /  \\  /  \\  /  \\  /  \\\n",
      "                /    \\/    \\/    \\/    \\\n",
      "███████████████/  /██/  /██/  /██/  /████████████████████████\n",
      "              /  / \\   / \\   / \\   / \\  \\____\n",
      "             /  /   \\_/   \\_/   \\_/   \\    o \\__,\n",
      "            / _/                       \\_____/  `\n",
      "            |/\n",
      "        ███╗   ███╗ █████╗ ███╗   ███╗██████╗  █████╗\n",
      "        ████╗ ████║██╔══██╗████╗ ████║██╔══██╗██╔══██╗\n",
      "        ██╔████╔██║███████║██╔████╔██║██████╔╝███████║\n",
      "        ██║╚██╔╝██║██╔══██║██║╚██╔╝██║██╔══██╗██╔══██║\n",
      "        ██║ ╚═╝ ██║██║  ██║██║ ╚═╝ ██║██████╔╝██║  ██║\n",
      "        ╚═╝     ╚═╝╚═╝  ╚═╝╚═╝     ╚═╝╚═════╝ ╚═╝  ╚═╝\n",
      "\n",
      "        mamba (0.15.3) supported by @QuantStack\n",
      "\n",
      "        GitHub:  https://github.com/mamba-org/mamba\n",
      "        Twitter: https://twitter.com/QuantStack\n",
      "\n",
      "█████████████████████████████████████████████████████████████\n",
      "\n",
      "\n",
      "Looking for: ['html5lib==1.1']\n",
      "\n",
      "pkgs/main/linux-64       Using cache\n",
      "pkgs/main/noarch         Using cache\n",
      "pkgs/r/linux-64          Using cache\n",
      "pkgs/r/noarch            Using cache\n",
      "\n",
      "Pinned packages:\n",
      "  - python 3.7.*\n",
      "\n",
      "\n",
      "Transaction\n",
      "\n",
      "  Prefix: /home/jupyterlab/conda/envs/python\n",
      "\n",
      "  Updating specs:\n",
      "\n",
      "   - html5lib==1.1\n",
      "   - ca-certificates\n",
      "   - certifi\n",
      "   - openssl\n",
      "\n",
      "\n",
      "  Package         Version  Build         Channel                 Size\n",
      "───────────────────────────────────────────────────────────────────────\n",
      "  Install:\n",
      "───────────────────────────────────────────────────────────────────────\n",
      "\n",
      "\u001b[32m  + html5lib    \u001b[00m      1.1  pyhd3eb1b0_0  pkgs/main/noarch       91 KB\n",
      "\u001b[32m  + webencodings\u001b[00m    0.5.1  py37_1        pkgs/main/linux-64     19 KB\n",
      "\n",
      "  Summary:\n",
      "\n",
      "  Install: 2 packages\n",
      "\n",
      "  Total download: 110 KB\n",
      "\n",
      "───────────────────────────────────────────────────────────────────────\n",
      "\n",
      "Downloading  [======>                                  ] (00m:00s)   75.47 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished webencodings                         (00m:00s)              19 KB     75 KB/s\n",
      "Downloading  [======>                                  ] (00m:00s)   75.47 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [======>                                  ] (00m:00s)   75.47 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [======>                                  ] (00m:00s)   75.47 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=========================================] (00m:00s)  432.74 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "\u001b[2A\u001b[0KFinished html5lib                             (00m:00s)              91 KB    357 KB/s\n",
      "Downloading  [=========================================] (00m:00s)  432.74 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=========================================] (00m:00s)  432.74 KB/s\n",
      "Extracting   [>                                                      ] (--:--) \n",
      "Downloading  [=========================================] (00m:00s)  432.74 KB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        1 / 2\n",
      "Downloading  [=========================================] (00m:00s)  432.74 KB/s\n",
      "Extracting   [====================>                    ] (00m:00s)        1 / 2\n",
      "Downloading  [=========================================] (00m:00s)  432.74 KB/s\n",
      "Extracting   [=========================================] (00m:00s)        2 / 2\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "Collecting lxml==4.6.4\n",
      "  Downloading lxml-4.6.4-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (6.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.3/6.3 MB\u001b[0m \u001b[31m84.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: lxml\n",
      "  Attempting uninstall: lxml\n",
      "    Found existing installation: lxml 4.9.1\n",
      "    Uninstalling lxml-4.9.1:\n",
      "      Successfully uninstalled lxml-4.9.1\n",
      "Successfully installed lxml-4.6.4\n"
     ]
    }
   ],
   "source": [
    "#!pip install pandas==1.3.3\n",
    "#!pip install requests==2.26.0\n",
    "!mamba install bs4==4.10.0 -y\n",
    "!mamba install html5lib==1.1 -y\n",
    "!pip install lxml==4.6.4\n",
    "#!pip install plotly==5.3.1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Example\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we must use the `request` library to downlaod the webpage, and extract the text. We will extract Netflix stock data <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/netflix_data_webpage.html\"\n",
    "\n",
    "data  = requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we must parse the text into html using `beautiful_soup`\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(data, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can turn the html table into a pandas dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "netflix_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "# First we isolate the body of the table which contains all the information\n",
    "# Then we loop through each row and find all the column values for each row\n",
    "for row in soup.find(\"tbody\").find_all('tr'):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    # Finally we append the data of each row to the table\n",
    "    netflix_data = netflix_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now print out the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78,560,600</td>\n",
       "      <td>528.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66,927,600</td>\n",
       "      <td>502.81</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111,573,300</td>\n",
       "      <td>513.47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90,183,900</td>\n",
       "      <td>521.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61,902,300</td>\n",
       "      <td>538.85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low   Close       Volume Adj Close\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21   78,560,600    528.21\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81   66,927,600    502.81\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47  111,573,300    513.47\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66   90,183,900    521.66\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85   61,902,300    538.85"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the pandas `read_html` function using the url\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_html_pandas_data = pd.read_html(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or we can convert the BeautifulSoup object to a string\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "read_html_pandas_data = pd.read_html(str(soup))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beacause there is only one table on the page, we just take the first table in the list returned\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close*</th>\n",
       "      <th>Adj Close**</th>\n",
       "      <th>Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jun 01, 2021</td>\n",
       "      <td>504.01</td>\n",
       "      <td>536.13</td>\n",
       "      <td>482.14</td>\n",
       "      <td>528.21</td>\n",
       "      <td>528.21</td>\n",
       "      <td>78560600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>May 01, 2021</td>\n",
       "      <td>512.65</td>\n",
       "      <td>518.95</td>\n",
       "      <td>478.54</td>\n",
       "      <td>502.81</td>\n",
       "      <td>502.81</td>\n",
       "      <td>66927600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Apr 01, 2021</td>\n",
       "      <td>529.93</td>\n",
       "      <td>563.56</td>\n",
       "      <td>499.00</td>\n",
       "      <td>513.47</td>\n",
       "      <td>513.47</td>\n",
       "      <td>111573300</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Mar 01, 2021</td>\n",
       "      <td>545.57</td>\n",
       "      <td>556.99</td>\n",
       "      <td>492.85</td>\n",
       "      <td>521.66</td>\n",
       "      <td>521.66</td>\n",
       "      <td>90183900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Feb 01, 2021</td>\n",
       "      <td>536.79</td>\n",
       "      <td>566.65</td>\n",
       "      <td>518.28</td>\n",
       "      <td>538.85</td>\n",
       "      <td>538.85</td>\n",
       "      <td>61902300</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date    Open    High     Low  Close* Adj Close**     Volume\n",
       "0  Jun 01, 2021  504.01  536.13  482.14  528.21      528.21   78560600\n",
       "1  May 01, 2021  512.65  518.95  478.54  502.81      502.81   66927600\n",
       "2  Apr 01, 2021  529.93  563.56  499.00  513.47      513.47  111573300\n",
       "3  Mar 01, 2021  545.57  556.99  492.85  521.66      521.66   90183900\n",
       "4  Feb 01, 2021  536.79  566.65  518.28  538.85      538.85   61902300"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "netflix_dataframe = read_html_pandas_data[0]\n",
    "\n",
    "netflix_dataframe.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Webscraping to Extract Stock Data Exercise\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `requests` library to download the webpage <https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html>. Save the text of the response as a variable named `html_data`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "url= \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-PY0220EN-SkillsNetwork/labs/project/amazon_data_webpage.html\"\n",
    "html_data  = requests.get(url).text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parse the html data using `beautiful_soup`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup1 = BeautifulSoup(html_data, 'html5lib')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 1</b> What is the content of the title attribute:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>Amazon.com, Inc. (AMZN) Stock Historical Prices &amp; Data - Yahoo Finance</title>\n",
      "Amazon.com, Inc. (AMZN) Stock Historical Prices & Data - Yahoo Finance\n"
     ]
    }
   ],
   "source": [
    "for title in soup1.find_all('title'):\n",
    "    print(title)\n",
    "    print(title.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using beautiful soup extract the table with historical share prices and store it into a dataframe named `amazon_data`. The dataframe should have columns Date, Open, High, Low, Close, Adj Close, and Volume. Fill in each variable with the correct data from the list `col`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "amazon_data = pd.DataFrame(columns=[\"Date\", \"Open\", \"High\", \"Low\", \"Close\", \"Volume\"])\n",
    "\n",
    "for row in soup1.find(\"tbody\").find_all(\"tr\"):\n",
    "    col = row.find_all(\"td\")\n",
    "    date = col[0].text\n",
    "    Open = col[1].text\n",
    "    high = col[2].text\n",
    "    low = col[3].text\n",
    "    close = col[4].text\n",
    "    adj_close = col[5].text\n",
    "    volume = col[6].text\n",
    "    \n",
    "    amazon_data = amazon_data.append({\"Date\":date, \"Open\":Open, \"High\":high, \"Low\":low, \"Close\":close, \"Adj Close\":adj_close, \"Volume\":volume}, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print out the first five rows of the `amazon_data` dataframe you created.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Date</th>\n",
       "      <th>Open</th>\n",
       "      <th>High</th>\n",
       "      <th>Low</th>\n",
       "      <th>Close</th>\n",
       "      <th>Volume</th>\n",
       "      <th>Adj Close</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Jan 01, 2021</td>\n",
       "      <td>3,270.00</td>\n",
       "      <td>3,363.89</td>\n",
       "      <td>3,086.00</td>\n",
       "      <td>3,206.20</td>\n",
       "      <td>71,528,900</td>\n",
       "      <td>3,206.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Dec 01, 2020</td>\n",
       "      <td>3,188.50</td>\n",
       "      <td>3,350.65</td>\n",
       "      <td>3,072.82</td>\n",
       "      <td>3,256.93</td>\n",
       "      <td>77,556,200</td>\n",
       "      <td>3,256.93</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nov 01, 2020</td>\n",
       "      <td>3,061.74</td>\n",
       "      <td>3,366.80</td>\n",
       "      <td>2,950.12</td>\n",
       "      <td>3,168.04</td>\n",
       "      <td>90,810,500</td>\n",
       "      <td>3,168.04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Oct 01, 2020</td>\n",
       "      <td>3,208.00</td>\n",
       "      <td>3,496.24</td>\n",
       "      <td>3,019.00</td>\n",
       "      <td>3,036.15</td>\n",
       "      <td>116,226,100</td>\n",
       "      <td>3,036.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sep 01, 2020</td>\n",
       "      <td>3,489.58</td>\n",
       "      <td>3,552.25</td>\n",
       "      <td>2,871.00</td>\n",
       "      <td>3,148.73</td>\n",
       "      <td>115,899,300</td>\n",
       "      <td>3,148.73</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Date      Open      High       Low     Close       Volume Adj Close\n",
       "0  Jan 01, 2021  3,270.00  3,363.89  3,086.00  3,206.20   71,528,900  3,206.20\n",
       "1  Dec 01, 2020  3,188.50  3,350.65  3,072.82  3,256.93   77,556,200  3,256.93\n",
       "2  Nov 01, 2020  3,061.74  3,366.80  2,950.12  3,168.04   90,810,500  3,168.04\n",
       "3  Oct 01, 2020  3,208.00  3,496.24  3,019.00  3,036.15  116,226,100  3,036.15\n",
       "4  Sep 01, 2020  3,489.58  3,552.25  2,871.00  3,148.73  115,899,300  3,148.73"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 2</b> What is the name of the columns of the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Date', 'Open', 'High', 'Low', 'Close', 'Volume', 'Adj Close'], dtype='object')"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Question 3</b> What is the `Open` of the last row of the amazon_data dataframe?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "60    656.29\n",
       "Name: Open, dtype: object"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "amazon_data.tail(1).Open"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h2>About the Authors:</h2> \n",
    "\n",
    "<a href=\"https://www.linkedin.com/in/joseph-s-50398b136/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDeveloperSkillsNetworkPY0220ENSkillsNetwork23455606-2021-01-01\">Joseph Santarcangelo</a> has a PhD in Electrical Engineering, his research focused on using machine learning, signal processing, and computer vision to determine how videos impact human cognition. Joseph has been working for IBM since he completed his PhD.\n",
    "\n",
    "Azim Hirjani\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Change Log\n",
    "\n",
    "| Date (YYYY-MM-DD) | Version | Changed By | Change Description |\n",
    "| ----------------- | ------- | ---------- | ------------------ |\n",
    "\n",
    "```\n",
    "| 2021-06-09       | 1.2     | Lakshmi Holla|Added URL in question 3 |\n",
    "```\n",
    "\n",
    "\\| 2020-11-10        | 1.1     | Malika Singla | Deleted the Optional part |\n",
    "\\| 2020-08-27        | 1.0     | Malika Singla | Added lab to GitLab       |\n",
    "\n",
    "<hr>\n",
    "\n",
    "## <h3 align=\"center\"> © IBM Corporation 2020. All rights reserved. <h3/>\n",
    "\n",
    "<p>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python",
   "language": "python",
   "name": "conda-env-python-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}