Remotive/Remotive_Post_daily_jobs_on_slack.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1ba829d8",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "<img width=\"10%\" alt=\"Naas\" src=\"https://landen.imgix.net/jtci2pxwjczr/assets/5ice39g4.png?w=160\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea66eab1",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "# Remotive - Post daily jobs on slack\n",
    "<a href=\"https://app.naas.ai/user-redirect/naas/downloader?url=https://raw.githubusercontent.com/jupyter-naas/awesome-notebooks/master/Remotive/Remotive_Post_daily_jobs_on_slack.ipynb\" target=\"_parent\"><img src=\"https://naasai-public.s3.eu-west-3.amazonaws.com/open_in_naas.svg\"/></a><br><br><a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=&template=template-request.md&title=Tool+-+Action+of+the+notebook+\">Template request</a> | <a href=\"https://github.com/jupyter-naas/awesome-notebooks/issues/new?assignees=&labels=bug&template=bug_report.md&title=Remotive+-+Post+daily+jobs+on+slack:+Error+short+description\">Bug report</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "983983f7",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Tags:** #remotive #jobs #slack #gsheet #naas_drivers #automation #opendata #text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30997e75-794b-4043-8597-4c8caeb7c8dc",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "**Author:** [Sanjeet Attili](https://www.linkedin.com/in/sanjeet-attili-760bab190/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8512f02d",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Input"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a627c44",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c6d2468",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "from datetime import datetime\n",
    "import time\n",
    "from naas_drivers import gsheet, slack\n",
    "import naas"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e8a99df",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup slack channel configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61bcb79c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "SLACK_TOKEN = \"xoxb-1481042297777-3085654341191-xxxxxxxxxxxxxxxxxxxxxxxxx\"\n",
    "SLACK_CHANNEL = \"05_work\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc95ed0e",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup sheet log data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8582ec1a",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "spreadsheet_id = \"1EBefhkbmqaXMZLRCiafabf6xxxxxxxxxxxxxxxxxxx\"\n",
    "sheet_name = \"SLACK_CHANNEL_POSTS\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efa54fb3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Setup Remotive"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a91d6d7",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "#### Get categories from Remotive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9701c8c3",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_remotejob_categories():\n",
    "    req_url = f\"https://remotive.io/api/remote-jobs/categories\"\n",
    "    res = requests.get(req_url)\n",
    "    try:\n",
    "        res.raise_for_status()\n",
    "    except requests.HTTPError as e:\n",
    "        return e\n",
    "    res_json = res.json()\n",
    "    \n",
    "    # Get categories\n",
    "    jobs = res_json.get('jobs')\n",
    "    return pd.DataFrame(jobs)\n",
    "\n",
    "df_categories = get_remotejob_categories()\n",
    "df_categories"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ba60b33",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "#### Enter your parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a846062",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "categories = ['data'] # Pick the list of categories in columns \"slug\"\n",
    "date_from = - 10 # Choose date difference in days from now => must be negative"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daeb9937",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Set the Scheduler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "061ba0bc",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "naas.scheduler.add(recurrence=\"0 9 * * *\")\n",
    "# # naas.scheduler.delete() # Uncomment this line to delete your scheduler if needed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1f18f73",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c3e77a8",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get the sheet log of jobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d58e5a78",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "df_jobs_log = gsheet.connect(spreadsheet_id).get(sheet_name=sheet_name)\n",
    "df_jobs_log"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2d770b8",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Get all jobs posted after timestamp_date\n",
    "\n",
    "All jobs posted after the date from will be fetched.<br>\n",
    "In summary, we can set the value, in seconds, of 'search_data_from' to fetch all jobs posted since this duration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "686484fb",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "REMOTIVE_DATETIME = \"%Y-%m-%dT%H:%M:%S\"\n",
    "NAAS_DATETIME = \"%Y-%m-%d %H:%M:%S\"\n",
    "\n",
    "def get_remotive_jobs_since(jobs, date):\n",
    "    ret = []\n",
    "    for job in jobs:\n",
    "        publication_date = datetime.strptime(job['publication_date'], REMOTIVE_DATETIME).timestamp()\n",
    "        if publication_date > date:\n",
    "            ret.append({\n",
    "                'URL': job['url'],\n",
    "                'TITLE': job['title'],\n",
    "                'COMPANY': job['company_name'],\n",
    "                'PUBLICATION_DATE': datetime.fromtimestamp(publication_date).strftime(NAAS_DATETIME)\n",
    "            })\n",
    "    return ret\n",
    "\n",
    "def get_category_jobs_since(category, date, limit):\n",
    "    url = f\"https://remotive.io/api/remote-jobs?category={category}&limit={limit}\"\n",
    "    res = requests.get(url)\n",
    "    if res.json()['jobs']:\n",
    "        publication_date = datetime.strptime(res.json()['jobs'][-1]['publication_date'], REMOTIVE_DATETIME).timestamp()\n",
    "        if len(res.json()['jobs']) < limit or date > publication_date:\n",
    "            print(f\"Jobs from catgory {category} fetched ✅\")\n",
    "            return get_remotive_jobs_since(res.json()['jobs'], date)\n",
    "        else:\n",
    "            return get_category_jobs_since(category, date, limit + 5)\n",
    "    return []\n",
    "\n",
    "def get_jobs_since(categories: list,\n",
    "                   date_from: int):\n",
    "    if date_from >= 0:\n",
    "        return(\"'date_from' must be negative. Please update your parameter.\")\n",
    "    # Transform datefrom int to\n",
    "    search_jobs_from = date_from * 24 * 60 * 60   # days in seconds\n",
    "    timestamp_date = time.time() + search_jobs_from\n",
    "\n",
    "    jobs = []\n",
    "    for category in categories:\n",
    "        jobs += get_category_jobs_since(category, timestamp_date, 5)\n",
    "    print(f'- All job since {datetime.fromtimestamp(timestamp_date)} have been fetched -')\n",
    "    return pd.DataFrame(jobs)\n",
    "\n",
    "df_jobs = get_jobs_since(categories, date_from=date_from)\n",
    "df_jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8af6a47c",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Remove duplicate jobs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e235e807",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "def remove_duplicates(df1, df2):\n",
    "    # Get jobs log\n",
    "    jobs_log = df1.URL.unique()\n",
    "    \n",
    "    # Exclude jobs already log from jobs\n",
    "    df2 = df2[~df2.URL.isin(jobs_log)]\n",
    "    return df2.sort_values(by=\"PUBLICATION_DATE\")\n",
    "\n",
    "df_new_jobs = remove_duplicates(df_jobs_log, df_jobs)\n",
    "df_new_jobs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05922c12",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "## Output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0092c5b0",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Add new jobs on the sheet log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b81d8bb1",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "gsheet.connect(spreadsheet_id).send(sheet_name=sheet_name,\n",
    "                                    data=df_new_jobs,\n",
    "                                    append=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bf4a159",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "source": [
    "### Send all jobs link to the slack channel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ed1daad",
   "metadata": {
    "papermill": {},
    "tags": []
   },
   "outputs": [],
   "source": [
    "if len(df_new_jobs) > 0:\n",
    "    for _, row in df_new_jobs.iterrows():\n",
    "        url = row.URL\n",
    "        slack.connect(SLACK_TOKEN).send(SLACK_CHANNEL, f\"<{url}>\")\n",
    "else:\n",
    "    print(\"Nothing to published in Slack !\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "naas": {
   "notebook_id": "fca0d3d6-4bbf-4c53-9e34-61293a1c0776",
   "notebook_path": "Remotive/Remotive_Post_daily_jobs_on_slack.ipynb"
  },
  "papermill": {
   "default_parameters": {},
   "environment_variables": {},
   "parameters": {},
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}