week_2/2.1_time_series_data/time_series_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 2.1: Time Series Data\n",
    "\n",
    "This lecture, we are going to learn about pandas' [time series & date functionality](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#time-series-date-functionality) by exploring a Google trends [dataset](https://www.kaggle.com/GoogleNewsLab/food-searches-on-google-since-2004) of popular food search terms.\n",
    "\n",
    "**Learning goals:**\n",
    "- List the main time classes and apis in pandas\n",
    "- Set a time index in a DataFrame\n",
    "- Select dates and date ranges from a time index\n",
    "- Pivot a stacked table\n",
    "- Explore time series data with data visualization\n",
    "- Shift a time series\n",
    "- Calculate a rolling statistic\n",
    "- Resample a time series\n",
    "- Interpolate missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction to Time Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Time Series is a sequence of data points indexed by _time_. In Pandas, there are **three** main classes related to time series. According to the [official documentation](https://pandas.pydata.org/docs/user_guide/timeseries.html#overview):\n",
    "\n",
    ">**Timestamp**: A specific date and time with timezone support  \n",
    "**Timedelta**: An absolute time duration  \n",
    "**Period**: A span of time defined by a point in time and its associated frequency  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Timestamp\n",
    "\n",
    "🕗 The pandas **[`Timestamp`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html)** represents a specific point in time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.738561Z",
     "start_time": "2022-01-31T16:42:53.318059Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "pd.Timestamp('2012-12-21')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Like most other pandas classes, it wraps an efficient **NumPy [dtype](https://docs.scipy.org/doc/numpy/reference/arrays.datetime.html#basic-datetimes)** with useful methods and apis.  \n",
    "For example, it exposes **[time components](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#time-date-components)**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.743435Z",
     "start_time": "2022-01-31T16:42:57.740327Z"
    }
   },
   "outputs": [],
   "source": [
    "dt = pd.Timestamp('2012-12-21')\n",
    "print(dt.dayofweek)\n",
    "print(dt.day_name())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Timestamp` constructor **parses many datetime representations**, including python `datetime`, NumPy `datetime64`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.749357Z",
     "start_time": "2022-01-31T16:42:57.745690Z"
    }
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import numpy as np\n",
    "\n",
    "# String representation:\n",
    "print(pd.Timestamp('2022-12-21'))\n",
    "\n",
    "# Time components\n",
    "print(pd.Timestamp(2012, 12, 21))\n",
    "\n",
    "# Python datetime object\n",
    "print(pd.Timestamp(datetime.datetime(2012, 12, 21)))\n",
    "\n",
    "# NumPy datetime64 object\n",
    "print(pd.Timestamp(np.datetime64('2012-12-21')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, pandas offers the convenient **[`.to_datetime()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_datetime.html)** method, which parses almost anything you'll throw at it. So no need to use `Timestamp` constructors directly!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.756591Z",
     "start_time": "2022-01-31T16:42:57.751563Z"
    }
   },
   "outputs": [],
   "source": [
    "print(pd.to_datetime('21-12-2012'))\n",
    "print(pd.to_datetime('2012-12-21'))\n",
    "print(pd.to_datetime('21st of December 2012'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Timedelta\n",
    "\n",
    "⏱ The pandas **[`Timedelta`](https://pandas.pydata.org/pandas-docs/stable/user_guide/timedeltas.html)** represents a **duration of time**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.761637Z",
     "start_time": "2022-01-31T16:42:57.758470Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.Timedelta('42 days 666 hours')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These durations can also be **negative**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.766266Z",
     "start_time": "2022-01-31T16:42:57.763090Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.Timedelta('-1 hr 3 min 3 s 7us')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like `Timestamp`, `Timedelta` **parses many data types**, including python `timedelta` and NumPy `timedelta64`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.771547Z",
     "start_time": "2022-01-31T16:42:57.767836Z"
    }
   },
   "outputs": [],
   "source": [
    "print(pd.Timedelta('42 days 666h'))\n",
    "print(pd.Timedelta(days=42, hours=666))\n",
    "print(pd.Timedelta(datetime.timedelta(days=42, hours=666)))\n",
    "print(pd.Timedelta(np.timedelta64(1, 'ms')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Timedelta`s are particularly useful to carry out **arithmetic operations** on `Timestamp`s. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.780029Z",
     "start_time": "2022-01-31T16:42:57.776270Z"
    }
   },
   "outputs": [],
   "source": [
    "day1 = pd.Timestamp('2012-12-21')\n",
    "print(f'The 21st of December 2012 was a {day1.day_name()}')\n",
    "\n",
    "day2 = day1 + pd.Timedelta('1 day')\n",
    "print(f'The day after was a {day2.day_name()}')\n",
    "\n",
    "td = pd.Timestamp.now() - pd.Timestamp('2012-12-21')\n",
    "print(f'It has been {td.days} days since the end of the world! 🙀')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Period"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "⏳ [`Period`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Period.html) represents **fixed-frequency intervals**.\n",
    "\n",
    "For example, let's make a period with one hour frequency, starting on the 21st of December 2012:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.786883Z",
     "start_time": "2022-01-31T16:42:57.782855Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.Period('2012-12-21', freq='h')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You don't always have to specify the frequency, but the Period object needs one.  \n",
    "In fact, pandas will **infer the frequency** based on the date format used as argument (be careful!):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.792825Z",
     "start_time": "2022-01-31T16:42:57.788948Z"
    }
   },
   "outputs": [],
   "source": [
    "print(repr(pd.Period('2011-01-01')))\n",
    "print(repr(pd.Period('2011-01')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Periods aren't very useful on their own, but shine when used as a TimeSeries _index_. More on this in [this section](#2.-Time-Indexing)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Time Indexing\n",
    "### Different types of Time indices\n",
    "\n",
    "The `Timestamp`, `Timedelta`, and `Period` classes are quite fun for manipulating dates and time intervals, but this is a lecture about _time series_ , i.e sequences of time data. For this, we need to create time indices. \n",
    "\n",
    "There are **three main types of time indices**, one for each of the time classes:\n",
    "- `DatetimeIndex` is a sequence of `Timestamp`\n",
    "- `TimedeltaIndex` is a sequence of `Timedelta`\n",
    "- `PeriodIndex` is a sequence of `Period`\n",
    "\n",
    "(More details in the [official documentation](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#overview))\n",
    "\n",
    "Let's create a `DatetimeIndex`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.799469Z",
     "start_time": "2022-01-31T16:42:57.794796Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.DatetimeIndex(['2020-01-01', '2020-01-04', '2020-01-05', '2024-03-09']) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, pandas offers a more convenient way to construct `DatetimeIndex`. Use `pd.date_range()` to create regularly spaced sequences of `Timestamp`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.806312Z",
     "start_time": "2022-01-31T16:42:57.801565Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.date_range('1969-07-20', periods=4, freq='H')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also do the same with `TimedeltaIndex` and `PeriodIndex`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.812571Z",
     "start_time": "2022-01-31T16:42:57.808032Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.timedelta_range(0, periods=4, freq='H')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.818470Z",
     "start_time": "2022-01-31T16:42:57.814097Z"
    }
   },
   "outputs": [],
   "source": [
    "pd.period_range('1969-07-20', periods=4, freq='H')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🧠 Notice that the `PeriodIndex` constructor looks exactly the same as the `DatetimeIndex` constructor... Can you explain the difference between these two types of indices?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adding a DateTimeIndex to a Series\n",
    "We've created time indices... Now let's use them in a `Series`!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.826813Z",
     "start_time": "2022-01-31T16:42:57.820408Z"
    }
   },
   "outputs": [],
   "source": [
    "index = pd.date_range('2000-01-01', periods=60, freq='D')\n",
    "ts = pd.Series(np.random.randn(len(index)), index=index)\n",
    "ts.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the series is \"aware\" of its index frequency, `D` (one day)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using the DateTimeIndex\n",
    "Since our time series is still a pandas `Series`, we can use all the indexing tricks learnt in lecture 1.2:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.832538Z",
     "start_time": "2022-01-31T16:42:57.828495Z"
    }
   },
   "outputs": [],
   "source": [
    "# selecting by index label\n",
    "ts['2000-01-05']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.837624Z",
     "start_time": "2022-01-31T16:42:57.834333Z"
    }
   },
   "outputs": [],
   "source": [
    "# selecting by index position\n",
    "ts[4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.844467Z",
     "start_time": "2022-01-31T16:42:57.839466Z"
    }
   },
   "outputs": [],
   "source": [
    "# list slicing 3rd element through to 5th element\n",
    "ts[2:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.850312Z",
     "start_time": "2022-01-31T16:42:57.846070Z"
    }
   },
   "outputs": [],
   "source": [
    "# list slicing elements in steps of 4\n",
    "ts[::4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Having a time index opens up many more possibilities with data selection. For example, we can..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.856112Z",
     "start_time": "2022-01-31T16:42:57.852001Z"
    }
   },
   "outputs": [],
   "source": [
    "# selecting by index label with datetime object\n",
    "ts[datetime.datetime(2000, 1, 5)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.862144Z",
     "start_time": "2022-01-31T16:42:57.858220Z"
    }
   },
   "outputs": [],
   "source": [
    "# selecting by index label with parsed datetime string\n",
    "ts['01/05/2000']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.869723Z",
     "start_time": "2022-01-31T16:42:57.863790Z"
    }
   },
   "outputs": [],
   "source": [
    "# selecting by slice of parsed datetime string\n",
    "ts['3rd of January 2000':'5th of January 2000']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.881207Z",
     "start_time": "2022-01-31T16:42:57.876221Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# selecting by range of parsed datetime string\n",
    "ts['February 2000']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, of course, all of this magic also applies to dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:57.894241Z",
     "start_time": "2022-01-31T16:42:57.883670Z"
    }
   },
   "outputs": [],
   "source": [
    "dft = pd.DataFrame(np.random.randn(666, 2), columns=['SAD', 'PEPE'], index=pd.date_range('20121221', periods=666, freq='D'))\n",
    "dft.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:58.683108Z",
     "start_time": "2022-01-31T16:42:57.895916Z"
    }
   },
   "outputs": [],
   "source": [
    "dft.loc['10th March 2013':'20th March 2013', :].plot.bar()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🧠 Take the time to understand what's happening in the two cells above. What's used as arguments to the `DataFrame` constructor? What is being selected with the `.loc[]` operator?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Time Series Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've conquered the basics of `TimeSeries`, let's put our skills to practice on some real data!\n",
    "\n",
    "The `food_searches.csv` dataset tracks the [Google trends](https://trends.google.com/trends) popularity of various foods and drinks from 2004 through 2016. Please bear in mind that most of this data is USA centric, so don't draw global conclusions on the trends just yet 🙃. \n",
    "\n",
    "Just like in lecture 1.2, let's start with some summary statistics to get insight into the values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:58.748166Z",
     "start_time": "2022-01-31T16:42:58.684808Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "df = pd.read_csv('food_searches.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:58.795724Z",
     "start_time": "2022-01-31T16:42:58.749721Z"
    }
   },
   "outputs": [],
   "source": [
    "df.describe(include='all')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Column description**  \n",
    "The `id` column holds the `string` name of the food/drink, `googleTopic` seems to be a google specific id, `week_id` is a date period, and `value` is the popularity of the search term. `value` has `min=0` and `max=100`, so we can expect a normalised value in percentage.\n",
    "\n",
    "Notice how the `DataFrame` index is **_not_ a time index**:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:58.800339Z",
     "start_time": "2022-01-31T16:42:58.797046Z"
    }
   },
   "outputs": [],
   "source": [
    "df.index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means we need to manually set the time index. However, **`week_id` isn't a time data type...**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:58.824224Z",
     "start_time": "2022-01-31T16:42:58.802012Z"
    }
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We don't want to end up with an `object` index! Then we won't be able to do all that fancy time indexing. So first things first: let's convert `week_id` to a datetime `dtype`. Remember how pandas makes date conversion easy with `pd.to_datetime()`? Well this `week_id` is particulary strange, so we've got help out the parsing a little with the **`format` argument**. Specifying date string formats is a very common occurence, a list of the symbols can be found in the python [documentation](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior).\n",
    "\n",
    "ℹ️ Weird formats happen all the time with real-world data, and that's partly why pandas exists! Knowing how to deal with these scenarios is a crucial data science skill."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:59.086431Z",
     "start_time": "2022-01-31T16:42:58.825847Z"
    }
   },
   "outputs": [],
   "source": [
    "# First add the day of the week to the `week_id` string\n",
    "df['datetime'] = df['week_id'] + '-1'\n",
    "# Then indicate the date string format\n",
    "df['datetime'] = pd.to_datetime(df['datetime'], format='%Y-%W-%w')\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:59.095126Z",
     "start_time": "2022-01-31T16:42:59.088039Z"
    }
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a `datetime64` column, we simply have to set it as our `DataFrame` index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:59.110225Z",
     "start_time": "2022-01-31T16:42:59.096229Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.set_index('datetime')\n",
    "df.loc['4th of July 2011']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! Now it will be much easier to explore the dataset with our shiny new time index. One thing still feels weird though... Ideally, we'd like to visualise and compare the trends of different foods and drinks. But in our `Dataframe`, those values are [\"stacked\"](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.stack.html) and differentiated using the column _value_ `id`. In previous lectures, when looking at different _features_ of a dataset, we had those separated into _columns_ , not groups of _rows_. And that made it easy to select, update, calculate, and plot those _features_. \n",
    "\n",
    "But fear not! Once again, pandas is here to rescue us. This is a common manipulation called \"reshaping\" or \"pivoting\" a table (more details in the [official documentation](https://pandas.pydata.org/docs/user_guide/reshaping.html)). We can disentangle this mess by using [`.pivot()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.pivot.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:59.176269Z",
     "start_time": "2022-01-31T16:42:59.111559Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.pivot(columns='id', values='value')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better! 😌 We can see some pesky `NaN`s though... they always sneak in our datasets! Just like in lecture 1.2, we want to get rid of them. They might ruin our beautiful plots! 🎨 \n",
    "\n",
    "However, we don't want to get rid of an entire _week_ of data if one food's popularity value is missing. Instead, we'd like to **get rid of the food _column_** if it contains any `NaN`. We can do this with the `axis` argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:59.202014Z",
     "start_time": "2022-01-31T16:42:59.177813Z"
    }
   },
   "outputs": [],
   "source": [
    "df = df.dropna(axis=1)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like `apple-ru` is gone! 🇷🇺That's okay, we still have 185 trends to analyse... Speaking of 🍎, let's visualise the popularity of `apple` searches on Google between 2004 and 2016:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:42:59.353638Z",
     "start_time": "2022-01-31T16:42:59.203783Z"
    }
   },
   "outputs": [],
   "source": [
    "df['apple'].plot.area()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That looks highly seasonal! [Seasonality](https://en.wikipedia.org/wiki/Seasonality) is the presence of regular variations in data. In this case, we expect that the seasonality of apples follows... seasons! It's not easy to see which months are the most red delicious from the graph. Trying to gauge seasonality is a common challenge with time series data. \n",
    "\n",
    "The solution is **seasonal plots**. Let's make one by splitting this graph by year, and plotting the yearly trends individually. We'll put the code in a function so we can explore the seasonality of many foods easily:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:45:39.891570Z",
     "start_time": "2022-01-31T16:45:39.884687Z"
    }
   },
   "outputs": [],
   "source": [
    "# new `year` column\n",
    "df['year'] = df.index.year\n",
    "# new week column\n",
    "df['week'] = df.index.isocalendar().week\n",
    "\n",
    "def plot_seasonal(df, food):\n",
    "    # pivot the dataframe so years are columns\n",
    "    pivot_df = df.pivot(index='week', columns='year', values=food)\n",
    "    # plot one line per year\n",
    "    pivot_df.plot.line(alpha=0.6, legend=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:45:40.448689Z",
     "start_time": "2022-01-31T16:45:40.265852Z"
    }
   },
   "outputs": [],
   "source": [
    "plot_seasonal(df, 'chocolate')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Peak analysis**  \n",
    "🍫 Chocolate sure is popular around holiday season! Why do you think there is a peak in mid-february? 💝 \n",
    "\n",
    "Let's explore the trends of a food slightly less fit for special occasions: donuts 🍩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:47:07.555979Z",
     "start_time": "2022-01-31T16:47:07.401608Z"
    }
   },
   "outputs": [],
   "source": [
    "plot_seasonal(df, 'donut')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, donuts are less festive than chocolate. I wonder what caused those peaks! Feel free to explore the data yourself 🤠\n",
    "\n",
    "Those differences in trends are interesting, but we'd like to compare them more closely. We have a suspicion that some alcoholic drinks have very different seasonal profiles... For example, `champagne` is usually popping around New Year's Eve, whilst `mojito` is a classic summer cocktail.\n",
    "\n",
    "**Average seasonal plots** allow us to compare the seasonality of two variables. Let's group the values by week, and plot the average values. The result is a graph of the average popularity of the drinks for each week. e.g the values of `champagne` at `week=20` is simply the mean of all the `champagne` values in df happening on the 20th week of the year."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:48:32.280578Z",
     "start_time": "2022-01-31T16:48:32.100621Z"
    }
   },
   "outputs": [],
   "source": [
    "df.groupby('week').mean().plot.line(y=['champagne', 'mojito'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🍾 Our instincts are confirmed: the New Year is a better time for bubbles than for rum!\n",
    "\n",
    "Sometimes, we wish to track the **cumulative trends** of some variables as well as their **relative trends**. For example, we might be interested in learning the popularity of _beverages_, as well as the breakdown into `tea`, `coffee`, and a hip newcomer `cold-brew-coffee`. \n",
    "\n",
    "**Area charts** allow to stack lines and observe the cumulative result. We're interested in long term trends, so we'll use the **yearly averages** to remove some of the distracting seasonal variations we observed earlier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:49:35.196023Z",
     "start_time": "2022-01-31T16:49:35.011672Z"
    }
   },
   "outputs": [],
   "source": [
    "df.groupby(df.index.year).mean()[['tea', 'coffee', 'cold-brew-coffee']].plot.area()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are three key insights from this graph:\n",
    "- beverages are on the rise since 2004\n",
    "- coffee got a boost around 2010\n",
    "- cold brews really took off around 2014\n",
    "\n",
    "ℹ️ This high \"density\" of information is key in effective data visualization, as explained in the legendary book [The Visual Display of Quantitative Information](https://www.goodreads.com/book/show/17744.The_Visual_Display_of_Quantitative_Information). More on this in lecture 2.3!\n",
    "\n",
    "💪 Create an area plot of the popularity of `ice-cream` from January to July 2006."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# INSERT YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🧠 Notice how the seasonality of your graph isn't obvious because of the chosen time window. Can you think of ways to avoid missing this detail in a time series analysis?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Shifting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The broccoli search expert at Google made a calendar mistake... All his **data is late by 8 weeks**! We don't want to manually calculate a new column and reindex our `DataFrame`. We _certainly_ don't want to manually update the values. Instead, we can use the [`.shift()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.shift.html) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:54:17.322720Z",
     "start_time": "2022-01-31T16:54:17.034219Z"
    }
   },
   "outputs": [],
   "source": [
    "# shift values by 8 weeks and save in new column\n",
    "df['apple_shift'] = df.shift(periods=-8*7, freq='D')['apple']\n",
    "# plot old and shifted data\n",
    "df.loc['2013', ['apple', 'apple_shift']].plot.line()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "More details in the [python data science handbook](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html#Time-shifts)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Windowing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🦐 We hear word that shrimps were highly fashionable around 2010, so we'd like to pinpoint this \"shrimp chic\" theory. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:59:22.038612Z",
     "start_time": "2022-01-31T16:59:21.886736Z"
    }
   },
   "outputs": [],
   "source": [
    "df['shrimp'].plot.line()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's a definite shellfish bump around 2009, but the [variance](https://en.wikipedia.org/wiki/Variance) of the popularity is muddling the graph. We could **\"smooth out\"** the line by plotting its moving average. These [rolling statistics](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html#Rolling-windows) can be calculated with pandas' [`.rolling()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.rolling.html#pandas.Series.rolling) method. The api interface is similar to [`.groupby`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html) (from lecture 1.3). The method returns a `Rolling` object, on which we must apply an _aggregation function_. In our case, we want a rolling _average_ , so we'll use the [`.mean()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.core.window.rolling.Rolling.mean.html) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T16:59:34.460185Z",
     "start_time": "2022-01-31T16:59:33.999829Z"
    }
   },
   "outputs": [],
   "source": [
    "# calculate rolling average of window size 12\n",
    "df['shrimp_av'] = df['shrimp'].rolling(12).mean()\n",
    "# plot at two different scales\n",
    "df.loc[:, ['shrimp', 'shrimp_av']].plot.line()\n",
    "df.loc['2013', ['shrimp', 'shrimp_av']].plot.line()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "📈 Each point of `shrimp_av` is the average popularity of the 12 previous weeks, and allows us to focus on longer term trends. \n",
    "\n",
    "💪 Create a line graph of the rolling standard deviation of the popularity of `coconut` 🥥 with a window size of 10, for the year 2012."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# INSERT YOUR CODE HERE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🧠 Can you explain what these peaks and troughs represent?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Resampling\n",
    "\n",
    "It is common to encounter datasets with irregular time indices, e.g a `DatetimeIndex` where the `Timestamp`s are not regularly spaced. This can be hard to work with for certain downstream tasks, such as times series prediction. One solution is to _resample_ the data to a regular interval. All the old values found in a new interval need to be combined using an _aggregation function_. In this sense, resampling is similar to [windowing](#5.-Windowing), except the aggregation is done on fixed frequency intervals instead of a sliding window. \n",
    "\n",
    "In pandas, resampling is done with [`.resample()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.resample.html). For example, we can resample our google trends to a monthly frequency:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T17:22:09.938930Z",
     "start_time": "2022-01-31T17:22:09.897389Z"
    }
   },
   "outputs": [],
   "source": [
    "df.resample('M').mean().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The time index now steps in months, and not in weeks. All the values in this new `DataFrame` were _averaged_. More information on resampling and converting frequencies in the [python data science handbook](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html#Resampling-and-converting-frequencies)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Interpolation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T17:23:16.217025Z",
     "start_time": "2022-01-31T17:23:16.212652Z"
    }
   },
   "outputs": [],
   "source": [
    "def forget_data(df, food):\n",
    "    np.random.seed(1337)\n",
    "    missing_indices = np.random.randint(low=0, high=len(df), size=42)\n",
    "    df.iloc[missing_indices, df.columns.get_loc(food)] = None\n",
    "    \n",
    "forget_data(df, 'long-island-iced-tea')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh no, someone had too many `long-island-iced-tea`s 🍹, and forgot some of the data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T17:23:18.171225Z",
     "start_time": "2022-01-31T17:23:17.922515Z"
    },
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df.loc['2012', 'long-island-iced-tea'].plot.line()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our beautiful graph! 😭 How do we repair such a mess? Instead of throwing away everything because of a few forgotten values, we can try to _interpolate_ the missing data. [Interpolation](https://en.wikipedia.org/wiki/Linear_interpolation) is \"guessing\" what the missing values are, based on their neighbours. Linear interpolation is the most common kind.\n",
    "\n",
    "💄 Let's make our graph pretty again with the [`.interpolate()`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.interpolate.html) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2022-01-31T17:23:52.653917Z",
     "start_time": "2022-01-31T17:23:52.405033Z"
    }
   },
   "outputs": [],
   "source": [
    "df['long-island-iced-tea'] = df['long-island-iced-tea'].interpolate()\n",
    "df.loc['2012', 'long-island-iced-tea'].plot.line()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Data repaired, hangover prevented. ⛑ Please note that the values are _not_ the same as before and there is no way to magically recover lost data. However, interpolation can enable a larger scale analysis that would otherwise fail, or act as a \"better than nothing\" solution. There are many interpolation methods, check out the [documentation](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Series.interpolate.html) for more details!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 8. Summary\n",
    "\n",
    "Today, we went on a tour of pandas' [time series & date functionality](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html#time-series-date-functionality). We learned about the **`Timestamp`**, **`Timedelta`**, and **`Period`** classes, and their associated time index classes. Then, we selected and manipulated a `DataFrame` using **time-based indexing**. We loaded a Google trends [dataset](https://www.kaggle.com/GoogleNewsLab/food-searches-on-google-since-2004) of popular food search terms, and **pivoted** the table to access our ordered time index. We explored the data, and visualized some of its time-specific aspects, such as **seasonality** and **comparative trends**. We also learned time data cleaning transformations, such as **index shifting**, **rolling statistics**, **resampling**, and **interpolation**. Overall, we discovered the main techniques for time series data exploration, tested them on a real dataset, and got insights into some dietary trends.\n",
    "\n",
    "# Resources\n",
    "## Core Resources\n",
    "\n",
    "- [Kaggle dataset of popular food searches](https://www.kaggle.com/GoogleNewsLab/food-searches-on-google-since-2004)\n",
    "- [Pandas time series & date functionality](https://pandas.pydata.org/pandas-docs/stable/user_guide/timeseries.html)\n",
    "- [Python Data Science Handbook - Time Series](https://jakevdp.github.io/PythonDataScienceHandbook/03.11-working-with-time-series.html)\n",
    "\n",
    "## Additional Resources\n",
    "\n",
    "- [Time formatting](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior)\n",
    "- [Exploring and visualizing time series](https://uc-r.github.io/ts_exploration)\n",
    "- [Times series analysis with pandas](https://www.dataquest.io/blog/tutorial-time-series-analysis-with-pandas/)\n",
    "- [Times series visualization with python](https://machinelearningmastery.com/time-series-data-visualization-with-python/)\n",
    "- [7 types of temporal visualizations](https://humansofdata.atlan.com/2016/11/visualizing-time-series-data/)\n",
    "- [11 stunning time series graphs](https://medium.com/@plotlygraphs/time-series-graphs-eleven-stunning-ways-you-can-use-them-cd1c1bcfe749)\n",
    "- [Reshaping and pivot tables](https://pandas.pydata.org/docs/user_guide/reshaping.html)\n",
    "- [Analysing time series data in pandas](https://towardsdatascience.com/analyzing-time-series-data-in-pandas-be3887fdd621)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}