week_5/5.1_learning_better_pt.1/learning_better_pt.1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lecture 5.1: Learning Better Pt.1\n",
    "\n",
    "This lecture, we are going to use feature scaling and polynomial features to improve the accuracy of a linear regression model.\n",
    "\n",
    "**Learning goals:**\n",
    "\n",
    "- scale a feature matrix using standardization\n",
    "- inverse transform scaled features to visualize the dataset in the original units\n",
    "- calculate polynomial features\n",
    "- use scaled polynomial features in a linear regression model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Covid-19 has been a test to our productivity and will power. 📱 In an effort to get our procrastination under control, we have decided to record data everytime we use instagram: the number of minutes we originally _planned_ to quickly check our dms, vs the _actual_ number of minutes we spent on that funny meme page.\n",
    "\n",
    "Our goal is to _predict_ the actual amount of time we will spend online in order to better organize our day. 📅 This is therefore a _regression_ task with:\n",
    "- one feature, the number of planned instagram minutes\n",
    "- a numerical label, the number of actual minutes spent on instagram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Feature Scaling\n",
    "\n",
    "Let's load the dataset from csv into a `DataFrame`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('instagram_planning.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This dataset only has two columns, so let's visualize it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.scatter(df['planned_minutes'], df['actual_minutes'], alpha=0.6)\n",
    "\n",
    "ax.set_xlabel('planned online time (min)')\n",
    "ax.set_ylabel('time spent online (min) ')\n",
    "ax.set_title('Online Procrastination');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this is where all our free time is going... The time spent online is considerably larger than the planned online time for each example.\n",
    "\n",
    "Notice how the x-axis ranges from 0 to 25 ? It seems that our feature has not been scaled. We can check with the `DataFrame`'s summary statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our feature, `planned_minutes`, is not _standardized_. Its mean is far from 0, and its standard deviation is greater than one. We have learned in the slides that feature scaling is almost always a good idea, so let's standardize this column.\n",
    "\n",
    "We'll use our beloved sklearn for this, and as always, its methods expect a feature _matrix_ , so let's [`.reshape()`](https://numpy.org/doc/stable/reference/generated/numpy.ndarray.reshape.html?highlight=reshape#numpy.ndarray.reshape) our single feature column into a $(30 \\times 1)$ matrix."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df['planned_minutes'].values.reshape(-1, 1)\n",
    "y = df['actual_minutes'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now able to `.fit()` our `StandardScaler` class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "scaler = StandardScaler().fit(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This might look weird: why is sklearn using the same `.fit()` api as learning _models_ , when this is just a simple equation to scale a dataset? \n",
    "\n",
    "Well if you recall from the slides, standardization updates each feature $x_{i}$ as follows:\n",
    "\n",
    "$$ \\textbf{x}_{i} := \\frac{\\textbf{x}_{i} - mean(\\textbf{x}_{i})}{std(\\textbf{x}_{i})}$$\n",
    "\n",
    "Observe how we need to calculate the _mean_ and _standard deviation_ of the feature across all the examples, in order to be able to standardize each value. This _interface_ is similar to how linear or logistic regression were trained: we passed the entire dataset as function argument. This is why sklearn chooses to reuse the name `.fit()` here, because it is consistent with the rest of its api where a class needs to be fed the dataset, `.fit(X, y)`, to operate. i.e `.fit()` is used when we have to adjust a class' behaviour to a dataset.\n",
    "\n",
    "Now that we have fit our standardizer to our feature, we can investigate the results. As explained above, the only two parameters that needed fitting here are the _mean_ and the _standard deviation_ of each feature. We can fetch these values as such:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler.mean_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler.scale_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore we now know that our trained `scaler` instance will standardize `planned_minutes` with the following transformation:\n",
    "\n",
    "$$ \\textbf{x}_{i} := \\frac{\\textbf{x}_{i} - 13.49}{5.58}$$\n",
    "\n",
    "And \"transformation\" is the correct term here: this is the name sklearn chose for the main method of its  `preprocessing` classes. We are indeed not _predicting_ anything here, but _updating_ the dataset. Without further a do:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_scaled = scaler.transform(X)\n",
    "\n",
    "# show the first 5 rows\n",
    "X_scaled[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.transform()` method returned our updated feature matrix, `X_scaled`. We can verify the standardization by visualizing the regression task once again:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.scatter(X_scaled, y, alpha=0.6)\n",
    "\n",
    "ax.set_xlabel('planned online time (norm)')\n",
    "ax.set_ylabel('time spent online (min)')\n",
    "ax.set_title('Online Procrastination');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `planned_minutes` feature, on the x-axis, has indeed been re-scaled.\n",
    "\n",
    "Notice that the y-axis hasn't changed, because it contains the _labels_ , and that feature scaling only applies to ... features!\n",
    "\n",
    "🧠 Can you think of why it is **not useful to scale labels**?\n",
    "\n",
    "Feature scaling is cool, and helps learning algorithms... but we have **lost the _interpretability_** of our visualization. What does 1 normalized unit of planned online time correspond to? 🤨 This graph is much harder to read! \n",
    "\n",
    "It is therefore important to know how to do _inverse_ the scaling applied to our features. That way, we can use the standardized values for learning, and change back the original units when reporting or visualizing the data.\n",
    "\n",
    "sklearn allows us to do this with `.inverse_transform()`. Remember how we previously `.fit()` our `StandardScaler`? It now remembers the mean and standard deviation of each feature, so it can reverse the standardization equation and recover the original values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_descaled = scaler.inverse_transform(X_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "ax.scatter(X_descaled, y, alpha=0.6)\n",
    "\n",
    "ax.set_xlabel('planned online time (min)')\n",
    "ax.set_ylabel('time spent online (min)')\n",
    "ax.set_title('Online Procrastination');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The unit of minutes of our `planned_minutes` was recovered!\n",
    "\n",
    "Granted, in this case, we still had access to the original `X` feature matrix, so re-calculating `X_descaled` wasn't super useful... 😅but knowing how to reverse preprocessing transformations in sklearn can be handy!\n",
    "\n",
    "And on the topic of usefulness: we learned in the slides that feature scaling was useful to _balance_ the features. This helped with the training of learning algorithms. However our dataset has only one feature... so it doesn't really need to be balanced against anything. Feature scaling in this example isn't particularly useful. However, it has given us an opportunity to discover the sklearn _preprocessing_ api. Which we will need for the next section..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## 3. Polynomial Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Training\n",
    "\n",
    "We wish to predict the actual amount of time we will spend online from the original `planned_minutes`. This is a regression task that can be solved using _linear regression_ :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "reg = LinearRegression().fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot the examples\n",
    "ax.scatter(X, y, alpha=0.6)\n",
    "\n",
    "# plot the hypothesis\n",
    "theta = np.append(reg.intercept_, reg.coef_)\n",
    "xmin, xmax = ax.get_xlim()\n",
    "x_line = np.linspace(xmin, xmax, 30)\n",
    "y_line = theta[0] + theta[1] * x_line\n",
    "ax.plot(x_line, y_line, c='g', linewidth=3)\n",
    "\n",
    "# formatting\n",
    "ax.set_xlim(xmin, xmax)\n",
    "ax.set_xlabel('planned online time (min)')\n",
    "ax.set_ylabel('time spent online (min)')\n",
    "ax.set_title('Online Procrastination');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our linear regression model correctly models the upwards trend in `actual_minutes`. However, it doesn't capture the slightly _parabolic_ nature of the data. i.e our model doesn't look very _accurate_.\n",
    "\n",
    "In order to fit this non-linear relationship between `actual_minutes` and `planned_minutes`, we must use a non-linear model. We have learned how to use _polynomial features_ to \"boost\" our linear regression hypothesis.\n",
    "\n",
    "sklearn offers polynomial regression under the form of another _preprocessor_ , [`PolynomialFeatures`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html). This uses same api as the `StandardScaler` from the [previous section](#2.-Feature-Scaling). It is in line with the `.fit()` and `.transform()` interface, since our polynomial features have to be calculated from the entire dataset, and are then added to the feature matrix.\n",
    "\n",
    "Let's apply polynomial features of degree 2 to our feature matrix `X`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "\n",
    "poly = PolynomialFeatures(2, include_bias=False)\n",
    "poly = poly.fit(X)\n",
    "X_poly = poly.transform(X)\n",
    "\n",
    "# show the first 5 rows\n",
    "X_poly[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our feature matrix, `X_poly`, now has two columns. The first column is our `planned_minutes`, the second is `planned_minutes` _squared_. We can verify this manually using the first row:\n",
    "\n",
    "$$ 2.618^{2} = 6.853 $$\n",
    "\n",
    "Which means that we now have polynomial features! This also means we have more than one feature. And as outlined in the [previous section](#2.-Feature-Scaling), features are best learned when they are _balanced_. We should therefore feature scale our new feature matrix `X_poly`.\n",
    "\n",
    "ℹ️ It is important to apply feature scaling _after_ polymial features are calculated. Scaled features will _not_ stay standardized when raised to the power of $n$. This is because exponents will change the standard deviation of a variable.\n",
    "\n",
    "We create a new `StandardScaler` instance since the previous one was already fitted to the original feature matrix `X`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "scaler = scaler.fit(X_poly)\n",
    "X_poly_scaled = scaler.transform(X_poly)\n",
    "\n",
    "# show the first 5 rows\n",
    "X_poly_scaled[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like our features values were indeed updated, but we can check that the standardization worked by looking at the mean and standard deviation of our columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x1 = X_poly_scaled[:, 0]\n",
    "x2 = X_poly_scaled[:, 1]\n",
    "\n",
    "print(f'feature 1 mean: {x1.mean()}, standard deviation: {x1.std()}')\n",
    "print(f'feature 2 mean: {x2.mean()}, standard deviation: {x2.std()}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are now ready for polynomial regression. We simply have to use our balanced polynomial features in a linear regression model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "reg = LinearRegression().fit(X_poly_scaled, y)\n",
    "\n",
    "theta = np.append(reg.intercept_, reg.coef_)\n",
    "theta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have learned 3 model parameters $\\theta$, as expected for two features. Since we know that the second feature is the square of the first feature, we can write our hypothesis as such:\n",
    "\n",
    "$$ h_{\\theta}(x) = 32.49 + 3.94 x + 16.94 x^{2} $$\n",
    "\n",
    "with $x$ our `planned_minutes` feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Prediction\n",
    "\n",
    "We can now use this trained regression model to predict the time we might spend online based on how long we were planning to browse instagram.\n",
    "\n",
    "Let's figure out how long we will spend online if we plan to browse instagram for $15$ minutes. We can make a feature matrix by sticking $15$ and $15^{2}$ in a NumPy `ndarray`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create feature matrix \n",
    "planned_minutes = 15\n",
    "X_predict = np.array([15, 15**2]).reshape(1, 2)\n",
    "\n",
    "# predict\n",
    "y_predict = reg.predict(X_predict)\n",
    "y_predict\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So if we plan to spend 15 minutes on instagram, we end up wasting... 3902 minutes?? 😰 That doesn't sound right. Indeed, we have forgotten to _scale our features_. Recall that we trained the regression model with scaled features to help its optimization process. Except now it's model parameters, $\\theta$, are going to be expecting _scaled_ data! Therefore we must always remember to apply the same transformations to the features we use for prediction, as the ones we used for training. This is super important, and is one of the most common bugs in machine learning 🙃\n",
    "\n",
    "Since we still have our fitted `scaler` and `poly` preprocessing instances, we can use both of them to apply all the feature transformations we need in one go. That way we don't have to manually calculate polynomial features, or apply the standardization equation, and most importantly we don't forget anything 😅.\n",
    "\n",
    "ℹ️ sklearn offers a convenient [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) api to stack preprocessors and models into one object. That way, it's even harder to forget some feature transformations!\n",
    "\n",
    "Let's try our prediction again, this time using our previously fitted preprocessors. As always, the sklearn api expects features _matrices_ , so we reshape the feature `ndarray` to a $(1 \\times 1)$ matrix from the very start:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create feature matrix\n",
    "planned_minutes = 15\n",
    "X_predict = np.array(15).reshape(1, 1)\n",
    "\n",
    "# apply feature transformations\n",
    "X_predict_poly = poly.transform(X_predict)\n",
    "X_predict_poly_scaled = scaler.transform(X_predict_poly)\n",
    "\n",
    "# predict\n",
    "y_predict = reg.predict(X_predict_poly_scaled)\n",
    "y_predict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "35 minutes spent online for 15 planned seems a more reasonable guess 😌."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Visualization\n",
    "\n",
    "To get a better overview of this model's predictions, we wish to visualize the non-linear hypothesis. Previously, we have used the values of $\\theta$ directly to calculate the predictions and plot them as a line:\n",
    "\n",
    "    x_line = np.linspace(-3, 3, 30)\n",
    "    y_line = theta[0] + theta[1] * x_line\n",
    "    ax.plot(x_line, y_line)\n",
    "   \n",
    "This creates an array of 30 points for our feature `x`, calculates the predictions directly using the hypothesis equation, then plots resulting line. \n",
    "\n",
    "However, we are now that we are dealing with feature scaling and high degree polynomials. So instead of manually calculating all these transformations and hypotheses on the features for predictions, we would like to leverage our fitted `reg` instance to predict the values of y. For this, we can use our preprocessors and the familiar `.predict()` api:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111)\n",
    "\n",
    "# plot the examples\n",
    "ax.scatter(X, y, alpha=0.6)\n",
    "\n",
    "# create feature matrix\n",
    "xmin, xmax = ax.get_xlim()\n",
    "x_line = np.linspace(xmin, xmax, 30).reshape(-1, 1)\n",
    "\n",
    "# apply feature transformations\n",
    "x_line_poly = poly.transform(x_line)\n",
    "x_line_poly_scaled = scaler.transform(x_line_poly)\n",
    "\n",
    "# predict\n",
    "y_line = reg.predict(x_line_poly_scaled)\n",
    "\n",
    "# plot the hypothesis\n",
    "ax.plot(x_line, y_line, c='g', linewidth=3)\n",
    "\n",
    "# formatting\n",
    "ax.set_xlim(xmin, xmax)\n",
    "ax.set_xlabel('planned online time (min)')\n",
    "ax.set_ylabel('time spent online (min)')\n",
    "ax.set_title('Online Procrastination');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a better fit for our data, and we can expect predictions with this polynomial regression model to be more accurate than the linear regression model. 🏹\n",
    "\n",
    "🧠🧠 Note that we are not plotting the preprocessed features `X_poly_scaled` and `x_line_poly_scaled` in this graph. Instead we use `X` and `x_line`. Why is that?\n",
    "\n",
    "Since adding polynomial features of degree 2 helped our regression model, perhaps adding degree 4 polynomials would be even better...\n",
    "\n",
    "💪💪 Add polynomial features of degree 4 to our dataset. Scale the resulting features with standardization. Then use the final feature matrix to train a polynomial regression model. The function to plot the hypothesis is provided and already called. To pass the unit test, please use the following naming convention:\n",
    "- `poly`: polynomial features preprocessor\n",
    "- `scaler`: standardization preprocessor\n",
    "- `reg`: regression model\n",
    "\n",
    "Pro-tip: start with the clean original feature matrix `X` or the sklearn preprocessors will get confused by the transformations of the previous cells"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# INSERT CODE HERE\n",
    "\n",
    "\n",
    "\n",
    "def plot_polynomial_regression(X, y, reg, poly, scaler):\n",
    "    fig = plt.figure()\n",
    "    ax = fig.add_subplot(111)\n",
    "\n",
    "    # plot the examples\n",
    "    ax.scatter(X, y, alpha=0.6)\n",
    "\n",
    "    # create feature matrix\n",
    "    xmin, xmax = ax.get_xlim()\n",
    "    x_line = np.linspace(xmin, xmax, 30).reshape(-1, 1)\n",
    "\n",
    "    # apply feature transformations\n",
    "    x_line_poly = poly.transform(x_line)\n",
    "    x_line_poly_scaled = scaler.transform(x_line_poly)\n",
    "\n",
    "    # predict\n",
    "    y_line = reg.predict(x_line_poly_scaled)\n",
    "\n",
    "    # plot the hypothesis\n",
    "    ax.plot(x_line, y_line, c='g', linewidth=3)\n",
    "\n",
    "    # formatting\n",
    "    ax.set_xlim(xmin, xmax)\n",
    "    ax.set_xlabel('planned online time (min)')\n",
    "    ax.set_ylabel('time spent online (min)')\n",
    "    ax.set_title('Online Procrastination');\n",
    "    \n",
    "import math\n",
    "\n",
    "def test_predict():\n",
    "    x_predict = np.array(25).reshape(1, 1)\n",
    "    x_predict_poly = poly.transform(x_predict)\n",
    "    x_predict_poly_scaled = scaler.transform(x_predict_poly)\n",
    "    y_predict = reg.predict(x_predict_poly_scaled)\n",
    "    assert math.isclose(y_predict, 85.01418, rel_tol=1e-4), \"The model isn't predicting the expected value\"\n",
    "    print('Success! 🎉')\n",
    "\n",
    "test_predict()\n",
    "plot_polynomial_regression(X, y, reg, poly, scaler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "🧠🧠 Does the degree 4 polynomial regression model look more accurate than the degree 2? Why do you think that is?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To go further\n",
    "Run this code to fit many polynomial regression models with increasing degrees and visualize the equation and the curve of their hypotheses.  \n",
    "At which point does increasing the degree become counterproductive?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "\n",
    "MAX_DEGREE = 15\n",
    "\n",
    "for degree in range(1, MAX_DEGREE):\n",
    "    pipeline = Pipeline([\n",
    "        (\"poly\", PolynomialFeatures(degree, include_bias=False)),\n",
    "        (\"scaler\", StandardScaler()),\n",
    "        (\"reg\", LinearRegression())\n",
    "    ])\n",
    "\n",
    "    pipe = pipeline.fit(X, y)\n",
    "    \n",
    "    \n",
    "    np.set_printoptions(precision=3)\n",
    "    intercept = f\"{pipe['reg'].intercept_:.2f}\"\n",
    "    coefs = [f\"{coef:.2f}*x^{i+1}\" for i, coef in enumerate(pipe[\"reg\"].coef_)]\n",
    "    \n",
    "    equation = \" + \".join([intercept] + coefs)\n",
    "    print(f\"Degree {degree}\")\n",
    "    print(f\"equation: {equation}\")\n",
    "    plot_polynomial_regression(X, y, pipe[\"reg\"], pipe[\"poly\"], pipe[\"scaler\"])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Summary\n",
    "\n",
    "Today, we have discovered two techniques to improve our learning algorithms, **feature scaling** and **polynomial features**. We highlighted that machine learning algorithms \"prefer\" **balanced** features, and that **standardization** is a method to make their mean and standard deviation uniform. We then understood the limitations of **linear models**, and learned how to calculate **polynomial features** to learn more complex hypotheses. Finally, we used these techniques to learn the **non-linear relationship** between the time we plan to spend online vs how long we end up procrastinating.\n",
    "\n",
    "# Resources\n",
    "\n",
    "## Core Resources\n",
    "\n",
    "- [sklearn - the importance of feature scaling](https://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html)  \n",
    "A visual example of the effect of feature scaling on PCA\n",
    "- [all about feature scaling](https://towardsdatascience.com/all-about-feature-scaling-bcc0ad75cb35)  \n",
    "Comprehensive breakdown of different feature scaling methods\n",
    "- [sklearn - polynomial features](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html)  \n",
    "sklearn official documentation on `PolynomialFeatures`\n",
    "- [sklearn - preprocessing data](https://scikit-learn.org/stable/modules/preprocessing.html)  \n",
    "All sklearn preprocessors in one place\n",
    "\n",
    "## Additional Resources\n",
    "\n",
    "- [standardization vs normalization](https://sebastianraschka.com/Articles/2014_about_feature_scaling.html)\n",
    "- [Compare the effect of different scalers on data with outliers](https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py)\n",
    "- [Quantitative comparison of different feature scaling methods on model accuracy](https://towardsdatascience.com/normalization-vs-standardization-quantitative-analysis-a91e8a79cebf)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}