Week 2 - Multiple Regression Assignment-1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regression Week 2: Multiple Regression (Interpretation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The goal of this first notebook is to explore multiple regression and feature engineering with existing graphlab functions.\n",
    "\n",
    "In this notebook you will use data on house sales in King County to predict prices using multiple regression. You will:\n",
    "* Use SFrames to do some feature engineering\n",
    "* Use built-in graphlab functions to compute the regression weights (coefficients/parameters)\n",
    "* Given the regression weights, predictors and outcome write a function to compute the Residual Sum of Squares\n",
    "* Look at coefficients and interpret their meanings\n",
    "* Evaluate multiple models via RSS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fire up graphlab create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import graphlab"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load in house sales data\n",
    "\n",
    "Dataset is from house sales in King County, the region where the city of Seattle, WA is located."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This non-commercial license of GraphLab Create for academic use is assigned to amitha353@gmail.com and will expire on May 07, 2019.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] graphlab.cython.cy_server: GraphLab Create v2.1 started. Logging: C:\\Users\\Amitha\\AppData\\Local\\Temp\\graphlab_server_1531506905.log.0\n"
     ]
    }
   ],
   "source": [
    "sales = graphlab.SFrame('kc_house_data.gl/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Split data into training and testing.\n",
    "We use seed=0 so that everyone running this notebook gets the same results.  In practice, you may set a random seed (or let GraphLab Create pick a random seed for you).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data,test_data = sales.random_split(.8,seed=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning a multiple regression model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall we can use the following code to learn a multiple regression model predicting 'price' based on the following features:\n",
    "example_features = ['sqft_living', 'bedrooms', 'bathrooms'] on training data with the following code:\n",
    "\n",
    "(Aside: We set validation_set = None to ensure that the results are always the same)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Linear regression:</pre>"
      ],
      "text/plain": [
       "Linear regression:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 17384</pre>"
      ],
      "text/plain": [
       "Number of examples          : 17384"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of features          : 3</pre>"
      ],
      "text/plain": [
       "Number of features          : 3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 3</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of coefficients    : 4</pre>"
      ],
      "text/plain": [
       "Number of coefficients    : 4"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Starting Newton Method</pre>"
      ],
      "text/plain": [
       "Starting Newton Method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |</pre>"
      ],
      "text/plain": [
       "| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 2        | 1.064115     | 4146407.600631     | 258679.804477 |</pre>"
      ],
      "text/plain": [
       "| 1         | 2        | 1.064115     | 4146407.600631     | 258679.804477 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>SUCCESS: Optimal solution found.</pre>"
      ],
      "text/plain": [
       "SUCCESS: Optimal solution found."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "example_features = ['sqft_living', 'bedrooms', 'bathrooms']\n",
    "example_model = graphlab.linear_regression.create(train_data, target = 'price', features = example_features, \n",
    "                                                  validation_set = None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have fitted the model we can extract the regression weights (coefficients) as an SFrame as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+-------+----------------+---------------+\n",
      "|     name    | index |     value      |     stderr    |\n",
      "+-------------+-------+----------------+---------------+\n",
      "| (intercept) |  None | 87910.0724924  |  7873.3381434 |\n",
      "| sqft_living |  None | 315.403440552  | 3.45570032585 |\n",
      "|   bedrooms  |  None | -65080.2155528 | 2717.45685442 |\n",
      "|  bathrooms  |  None | 6944.02019265  | 3923.11493144 |\n",
      "+-------------+-------+----------------+---------------+\n",
      "[4 rows x 4 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "example_weight_summary = example_model.get(\"coefficients\")\n",
    "print example_weight_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Predictions\n",
    "\n",
    "In the gradient descent notebook we use numpy to do our regression. In this book we will use existing graphlab create functions to analyze multiple regressions. \n",
    "\n",
    "Recall that once a model is built we can use the .predict() function to find the predicted values for data we pass. For example using the example model above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "271789.505878\n"
     ]
    }
   ],
   "source": [
    "example_predictions = example_model.predict(train_data)\n",
    "print example_predictions[0] # should be 271789.505878"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Compute RSS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we can make predictions given the model, let's write a function to compute the RSS of the model. Complete the function below to calculate RSS given the model, data, and the outcome."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_residual_sum_of_squares(model, data, outcome):\n",
    "    # First get the predictions\n",
    "    predictions = model.predict(data)\n",
    "    # Then compute the residuals/errors\n",
    "    residuals = outcome - predictions\n",
    "    # Then square and add them up\n",
    "    RSS = (residuals * residuals).sum()\n",
    "    return(RSS)    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test your function by computing the RSS on TEST data for the example model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.7376153833e+14\n"
     ]
    }
   ],
   "source": [
    "rss_example_train = get_residual_sum_of_squares(example_model, test_data, test_data['price'])\n",
    "print rss_example_train # should be 2.7376153833e+14"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create some new features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although we often think of multiple regression as including multiple different features (e.g. # of bedrooms, squarefeet, and # of bathrooms) but we can also consider transformations of existing features e.g. the log of the squarefeet or even \"interaction\" features such as the product of bedrooms and bathrooms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will use the logarithm function to create a new feature. so first you should import it from the math library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next create the following 4 new features as column in both TEST and TRAIN data:\n",
    "* bedrooms_squared = bedrooms\\*bedrooms\n",
    "* bed_bath_rooms = bedrooms\\*bathrooms\n",
    "* log_sqft_living = log(sqft_living)\n",
    "* lat_plus_long = lat + long \n",
    "As an example here's the first one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_data['bedrooms_squared'] = train_data['bedrooms'].apply(lambda x: x**2)\n",
    "test_data['bedrooms_squared'] = test_data['bedrooms'].apply(lambda x: x**2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# create the remaining 3 features in both TEST and TRAIN data\n",
    "train_data['bed_bath_rooms'] = train_data['bedrooms'] * train_data['bathrooms']\n",
    "test_data['bed_bath_rooms'] = test_data['bedrooms'] * test_data['bathrooms']\n",
    "\n",
    "train_data['log_sqft_living'] = train_data['sqft_living'].apply(lambda x: log(x))\n",
    "test_data['log_sqft_living'] = test_data['sqft_living'].apply(lambda x: log(x))\n",
    "\n",
    "train_data['lat_plus_long'] = train_data['lat'] + train_data['long']\n",
    "test_data['lat_plus_long'] = test_data['lat'] + test_data['long']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Squaring bedrooms will increase the separation between not many bedrooms (e.g. 1) and lots of bedrooms (e.g. 4) since 1^2 = 1 but 4^2 = 16. Consequently this feature will mostly affect houses with many bedrooms.\n",
    "* bedrooms times bathrooms gives what's called an \"interaction\" feature. It is large when *both* of them are large.\n",
    "* Taking the log of squarefeet has the effect of bringing large values closer together and spreading out small values.\n",
    "* Adding latitude to longitude is totally non-sensical but we will do it anyway (you'll see why)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question: What is the mean (arithmetic average) value of your 4 new features on TEST data? (round to 2 digits)**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "12.4466777016\n",
      "7.50390163159\n",
      "7.55027467965\n",
      "-74.6533349722\n"
     ]
    }
   ],
   "source": [
    "print test_data['bedrooms_squared'].mean()\n",
    "print test_data['bed_bath_rooms'].mean()\n",
    "print test_data['log_sqft_living'].mean()\n",
    "print test_data['lat_plus_long'].mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Learning Multiple Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will learn the weights for three (nested) models for predicting house prices. The first model will have the fewest features the second model will add one more feature and the third will add a few more:\n",
    "* Model 1: squarefeet, # bedrooms, # bathrooms, latitude & longitude\n",
    "* Model 2: add bedrooms\\*bathrooms\n",
    "* Model 3: Add log squarefeet, bedrooms squared, and the (nonsensical) latitude + longitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model_1_features = ['sqft_living', 'bedrooms', 'bathrooms', 'lat', 'long']\n",
    "model_2_features = model_1_features + ['bed_bath_rooms']\n",
    "model_3_features = model_2_features + ['bedrooms_squared', 'log_sqft_living', 'lat_plus_long']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have the features, learn the weights for the three different models for predicting target = 'price' using graphlab.linear_regression.create() and look at the value of the weights/coefficients:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre>Linear regression:</pre>"
      ],
      "text/plain": [
       "Linear regression:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 17384</pre>"
      ],
      "text/plain": [
       "Number of examples          : 17384"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of features          : 5</pre>"
      ],
      "text/plain": [
       "Number of features          : 5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 5</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of coefficients    : 6</pre>"
      ],
      "text/plain": [
       "Number of coefficients    : 6"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Starting Newton Method</pre>"
      ],
      "text/plain": [
       "Starting Newton Method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |</pre>"
      ],
      "text/plain": [
       "| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 2        | 0.124095     | 4074878.213096     | 236378.596455 |</pre>"
      ],
      "text/plain": [
       "| 1         | 2        | 0.124095     | 4074878.213096     | 236378.596455 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>SUCCESS: Optimal solution found.</pre>"
      ],
      "text/plain": [
       "SUCCESS: Optimal solution found."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Linear regression:</pre>"
      ],
      "text/plain": [
       "Linear regression:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 17384</pre>"
      ],
      "text/plain": [
       "Number of examples          : 17384"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of features          : 6</pre>"
      ],
      "text/plain": [
       "Number of features          : 6"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 6</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 6"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of coefficients    : 7</pre>"
      ],
      "text/plain": [
       "Number of coefficients    : 7"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Starting Newton Method</pre>"
      ],
      "text/plain": [
       "Starting Newton Method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |</pre>"
      ],
      "text/plain": [
       "| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 2        | 0.166113     | 4014170.932928     | 235190.935428 |</pre>"
      ],
      "text/plain": [
       "| 1         | 2        | 0.166113     | 4014170.932928     | 235190.935428 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>SUCCESS: Optimal solution found.</pre>"
      ],
      "text/plain": [
       "SUCCESS: Optimal solution found."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Linear regression:</pre>"
      ],
      "text/plain": [
       "Linear regression:"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of examples          : 17384</pre>"
      ],
      "text/plain": [
       "Number of examples          : 17384"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of features          : 9</pre>"
      ],
      "text/plain": [
       "Number of features          : 9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of unpacked features : 9</pre>"
      ],
      "text/plain": [
       "Number of unpacked features : 9"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Number of coefficients    : 10</pre>"
      ],
      "text/plain": [
       "Number of coefficients    : 10"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>Starting Newton Method</pre>"
      ],
      "text/plain": [
       "Starting Newton Method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>--------------------------------------------------------</pre>"
      ],
      "text/plain": [
       "--------------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |</pre>"
      ],
      "text/plain": [
       "| Iteration | Passes   | Elapsed Time | Training-max_error | Training-rmse |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>| 1         | 2        | 0.050013     | 3193229.177908     | 228200.043155 |</pre>"
      ],
      "text/plain": [
       "| 1         | 2        | 0.050013     | 3193229.177908     | 228200.043155 |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>+-----------+----------+--------------+--------------------+---------------+</pre>"
      ],
      "text/plain": [
       "+-----------+----------+--------------+--------------------+---------------+"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre>SUCCESS: Optimal solution found.</pre>"
      ],
      "text/plain": [
       "SUCCESS: Optimal solution found."
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre></pre>"
      ],
      "text/plain": []
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Learn the three models: (don't forget to set validation_set = None)\n",
    "model_1 = graphlab.linear_regression.create(train_data, target='price',features=model_1_features, validation_set=None)\n",
    "model_2 = graphlab.linear_regression.create(train_data, target='price',features=model_2_features, validation_set=None)\n",
    "model_3 = graphlab.linear_regression.create(train_data, target='price',features=model_3_features, validation_set=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+-------------+-------+----------------+---------------+\n",
      "|     name    | index |     value      |     stderr    |\n",
      "+-------------+-------+----------------+---------------+\n",
      "| (intercept) |  None | -56140675.7444 | 1649985.42028 |\n",
      "| sqft_living |  None | 310.263325778  | 3.18882960408 |\n",
      "|   bedrooms  |  None | -59577.1160683 | 2487.27977322 |\n",
      "|  bathrooms  |  None | 13811.8405419  | 3593.54213297 |\n",
      "|     lat     |  None | 629865.789485  | 13120.7100323 |\n",
      "|     long    |  None | -214790.285186 | 13284.2851607 |\n",
      "+-------------+-------+----------------+---------------+\n",
      "[6 rows x 4 columns]\n",
      "\n",
      "+----------------+-------+----------------+---------------+\n",
      "|      name      | index |     value      |     stderr    |\n",
      "+----------------+-------+----------------+---------------+\n",
      "|  (intercept)   |  None | -54410676.1152 | 1650405.16541 |\n",
      "|  sqft_living   |  None | 304.449298056  | 3.20217535637 |\n",
      "|    bedrooms    |  None | -116366.04323  | 4805.54966546 |\n",
      "|   bathrooms    |  None | -77972.3305131 | 7565.05991091 |\n",
      "|      lat       |  None | 625433.834953  | 13058.3530972 |\n",
      "|      long      |  None | -203958.602959 | 13268.1283711 |\n",
      "| bed_bath_rooms |  None | 26961.6249091  | 1956.36561555 |\n",
      "+----------------+-------+----------------+---------------+\n",
      "[7 rows x 4 columns]\n",
      "\n",
      "+------------------+-------+----------------+---------------+\n",
      "|       name       | index |     value      |     stderr    |\n",
      "+------------------+-------+----------------+---------------+\n",
      "|   (intercept)    |  None | -52974974.0598 | 1615194.94383 |\n",
      "|   sqft_living    |  None | 529.196420561  | 7.69913498509 |\n",
      "|     bedrooms     |  None | 28948.5277291  | 9395.72889104 |\n",
      "|    bathrooms     |  None | 65661.2072295  | 10795.3380703 |\n",
      "|       lat        |  None | 704762.148378  |      nan      |\n",
      "|       long       |  None | -137780.019966 |      nan      |\n",
      "|  bed_bath_rooms  |  None | -8478.36410481 | 2858.95391257 |\n",
      "| bedrooms_squared |  None | -6072.38466052 | 1494.97042777 |\n",
      "| log_sqft_living  |  None | -563467.78426  | 17567.8230813 |\n",
      "|  lat_plus_long   |  None | -83217.197896  |      nan      |\n",
      "+------------------+-------+----------------+---------------+\n",
      "[10 rows x 4 columns]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Examine/extract each model's coefficients:\n",
    "model_1_coefficients = model_1.get('coefficients')\n",
    "model_2_coefficients = model_2.get('coefficients')\n",
    "model_3_coefficients = model_3.get('coefficients')\n",
    "print model_1_coefficients\n",
    "print model_2_coefficients\n",
    "print model_3_coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 1?**\n",
    "+ (Positive) -> 13811.8405419\n",
    "\n",
    "**Quiz Question: What is the sign (positive or negative) for the coefficient/weight for 'bathrooms' in model 2?**\n",
    "- (Negative) -> -77972.3305131\n",
    "Think about what this means."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Comparing multiple models\n",
    "\n",
    "Now that you've learned three models and extracted the model weights we want to evaluate which model is best."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First use your functions from earlier to compute the RSS on TRAINING Data for each of the three models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "9.71328233544e+14\n",
      "9.61592067856e+14\n",
      "9.05276314556e+14\n"
     ]
    }
   ],
   "source": [
    "# Compute the RSS on TRAINING data for each of the three models and record the values:\n",
    "rss_model_1_train = get_residual_sum_of_squares(model_1, train_data, train_data['price'])\n",
    "rss_model_2_train = get_residual_sum_of_squares(model_2, train_data, train_data['price'])\n",
    "rss_model_3_train = get_residual_sum_of_squares(model_3, train_data, train_data['price'])\n",
    "print rss_model_1_train\n",
    "print rss_model_2_train\n",
    "print rss_model_3_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question: Which model (1, 2 or 3) has lowest RSS on TRAINING Data?** Is this what you expected?\n",
    "* Model 3 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now compute the RSS on on TEST data for each of the three models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.26568089093e+14\n",
      "2.24368799994e+14\n",
      "2.5182931895e+14\n"
     ]
    }
   ],
   "source": [
    "# Compute the RSS on TESTING data for each of the three models and record the values:\n",
    "rss_model_1_test = get_residual_sum_of_squares(model_1, test_data, test_data['price'])\n",
    "rss_model_2_test = get_residual_sum_of_squares(model_2, test_data, test_data['price'])\n",
    "rss_model_3_test = get_residual_sum_of_squares(model_3, test_data, test_data['price'])\n",
    "print rss_model_1_test\n",
    "print rss_model_2_test\n",
    "print rss_model_3_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Quiz Question: Which model (1, 2 or 3) has lowest RSS on TESTING Data?** Is this what you expected? Think about the features that were added to each model from the previous.\n",
    "* Model 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Quiz Solutions:\n",
    "<img src=\"quiz-w2-assignment-1-1,2,3,4.PNG\">\n",
    "<img src=\"quiz-w2-assignment-1-5,6,7,8.PNG\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}