2.a-Collect Data (API).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"text-align:center\">\n",
    "    <a href=\"https://skills.network/?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2022-01-01\" target=\"_blank\">\n",
    "    <img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/assets/logos/SN_web_lightmode.png\" width=\"200\" alt=\"Skills Network Logo\"  />\n",
    "    </a>\n",
    "</p>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Collecting Data Using APIs**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*   Collect job data from Jobs API\n",
    "*   Store the collected data into an excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step by step\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To run this notebook, firstly we need to click on the [Jobs_API](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Jobs_API.ipynb) notebook link, which is found in this same Github repository. The file contains flask code which is required to run the Jobs API data.\n",
    "\n",
    "Now, to run the code in the file that opens up we need to follow the below steps.\n",
    "\n",
    "Step1: We download the file.\n",
    "\n",
    "Step2: We upload it on the IBM Watson studio. (If IBM Watson Cloud service does not work in our system, we can follow the alternate Step 2 below)\n",
    "\n",
    "Step2(alternate): We upload it in oour SN labs environment using the upload button which is highlighted in red in the image below:\n",
    "We need to remember to upload this Jobs_API file in the same folder as oour current .ipynb file\n",
    "\n",
    "<img src=\"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/Upload.PNG\" />\n",
    "\n",
    "Step3:  We run all the cells of the Jobs_API file. (Even if we receive an asterik sign after running the last cell, the code works fine.)\n",
    "\n",
    "If you want to learn more about flask, which is optional, you can click on this link [here](https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/FLASK_API.md.html).\n",
    "\n",
    "Once we run the flask code, we can start with the fllowing instructions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset Used in this Assignment\n",
    "\n",
    "The dataset used in this notebook comes from the following source: [https://www.kaggle.com/promptcloud/jobs-on-naukricom](https://www.kaggle.com/promptcloud/jobs-on-naukricom?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2022-01-01) under the under a **Public Domain license**.\n",
    "\n",
    "\n",
    "The original dataset is a csv. We already converted the csv to json."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collecting Jobs Data using Jobs API\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to determine the number of jobs currently open for various technologies  and for various locations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we will collect the number of job postings for the following locations using the API:\n",
    "\n",
    "*   Los Angeles\n",
    "*   New York\n",
    "*   San Francisco\n",
    "*   Washington DC\n",
    "*   Seattle\n",
    "*   Austin\n",
    "*   Detroit\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We import required libraries\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We write a function to get the number of jobs for the Python technology.<br>\n",
    "\n",
    "\n",
    "##### The keys in the json are\n",
    "\n",
    "*   Job Title\n",
    "\n",
    "*   Job Experience Required\n",
    "\n",
    "*   Key Skills\n",
    "\n",
    "*   Role Category\n",
    "\n",
    "*   Location\n",
    "\n",
    "*   Functional Area\n",
    "\n",
    "*   Industry\n",
    "\n",
    "*   Role\n",
    "\n",
    "We can also view  the json file contents  from the following <a href = \"https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBM-DA0321EN-SkillsNetwork/labs/module%201/Accessing%20Data%20Using%20APIs/jobs.json?utm_medium=Exinfluencer&utm_source=Exinfluencer&utm_content=000026UJ&utm_term=10006555&utm_id=NA-SkillsNetwork-Channel-SkillsNetworkCoursesIBMDA0321ENSkillsNetwork21426264-2022-01-01\">json</a> URL.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_url=\"http://127.0.0.1:5000/data/all\"\n",
    "def get_number_of_jobs_T(technology):\n",
    "    \n",
    "    response_api = requests.get(api_url)\n",
    "\n",
    "    number_of_jobs = 0\n",
    "\n",
    "    if response_api.ok:            \n",
    "        jobs = response_api.json()\n",
    "\n",
    "    for job in jobs:\n",
    "        key = job.get('Key Skills')\n",
    "\n",
    "        if key.find(technology) > -1 :\n",
    "            number_of_jobs = number_of_jobs + 1\n",
    "\n",
    "    number_of_jobs\n",
    "    \n",
    "    return technology,number_of_jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's call the function for Python and checking if it works.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Python', 1173)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_number_of_jobs_T(\"Python\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We write a function to find number of jobs in US for a location of your choice\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_number_of_jobs_L(location):\n",
    "    \n",
    "    response_api = requests.get(api_url)\n",
    "\n",
    "    number_of_jobs = 0\n",
    "\n",
    "    if response_api.ok:            \n",
    "        jobs = response_api.json()\n",
    "\n",
    "    for job in jobs:\n",
    "        loc = job.get('Location')\n",
    "\n",
    "        if loc.find(location) > -1 :\n",
    "            number_of_jobs = number_of_jobs + 1\n",
    "\n",
    "    number_of_jobs\n",
    "    return location,number_of_jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let' call the function for Los Angeles and check if it is working.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('Los Angeles', 640)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_number_of_jobs_L(\"Los Angeles\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Storing the results in an excel file\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We call the API for all the given technologies above and write the results in an excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a python list of all locations for which you need to find the number of jobs postings.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "locations = ['Los Angeles','New York','San Francisco','Washington DC','Seattle','Austin','Detroit']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We import libraries required to create excel spreadsheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from openpyxl import Workbook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We create a workbook and select the active worksheet\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb1 = Workbook()\n",
    "ws1 = wb1.active"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We find the number of jobs postings for each of the location in the above list and write the Location name and the number of jobs postings into the excel spreadsheet.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws1.append(['Location','Number of Jobs'])\n",
    "\n",
    "for i in range(len(locations)):\n",
    "    ws1.append(get_number_of_jobs_L(locations[i]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We save into an excel spreadsheet named 'job-postings.xlsx'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb1.save('2.a-job-postings (Collected from API).xlsx')\n",
    "wb1.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the similar way, we can try for below given technologies and results  can be stored in an excel sheet.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We collect the number of job postings for the following languages using the API:\n",
    "\n",
    "*   C\n",
    "*   C#\n",
    "*   C++\n",
    "*   Java\n",
    "*   JavaScript\n",
    "*   Python\n",
    "*   Scala\n",
    "*   Oracle\n",
    "*   SQL Server\n",
    "*   MySQL Server\n",
    "*   PostgreSQL\n",
    "*   MongoDB\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "wb2 = Workbook()\n",
    "ws2 = wb2.active\n",
    "\n",
    "languages = ['C','C#','C++','Java','JavaScript','Python','Scala','Oracle','SQL Server','MySQL Server','PostgreSQL','MongoDB']\n",
    "\n",
    "ws2.append(['Languages','Number of Jobs'])\n",
    "\n",
    "for i in range(len(languages)):\n",
    "    ws2.append(get_number_of_jobs_T(languages[i]))\n",
    "\n",
    "wb2.save('2.a-job-postings-languages (Collected from API).xlsx')\n",
    "wb2.close()\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}