Skip to content

Commit 11ca594

Browse files
sandipanpaul21sandipanpaul21
committed
Created using Colaboratory
1 parent fc07942 commit 11ca594

File tree

1 file changed

+121
-28
lines changed

1 file changed

+121
-28
lines changed

‎07_Simple_and_Multiple_Regression.ipynb

Lines changed: 121 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
"name": "07 Simple and Multiple Regression.ipynb",
77
"provenance": [],
88
"collapsed_sections": [],
9-
"authorship_tag": "ABX9TyP3daddsjDhXZ17CoSmuNyw",
9+
"authorship_tag": "ABX9TyOAJPmXXYrhwvtGuE3Xfsil",
1010
"include_colab_link": true
1111
},
1212
"kernelspec": {
@@ -25,14 +25,109 @@
2525
"<a href=\"https://colab.research.google.com/github/sandipanpaul21/Machine-Learning-in-Python-Code/blob/master/07_Simple_and_Multiple_Regression.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
2626
]
2727
},
28+
{
29+
"cell_type": "markdown",
30+
"metadata": {
31+
"id": "O1w94qbJnRld"
32+
},
33+
"source": [
34+
"**LINEAR REGRESSION**\n",
35+
"\n",
36+
"Linear regression is studied as a model for understanding the relationship between input and output numerical variables.\n",
37+
"\n",
38+
"**Linear regression is a linear model**, e.g. a model that assumes a linear relationship between the input variables (x) and the single output variable (y). More specifically, that y can be calculated from a linear combination of the input variables (x).\n",
39+
"\n",
40+
"When there is a single input variable (x), the method is referred to as **simple linear regression**. When there are multiple input variables, literature from statistics often refers to the method as **multiple linear regression**\n",
41+
"\n",
42+
"Different techniques can be used to prepare or train the linear regression equation from data, the most common of which is called **Ordinary Least Squares**. It is common to therefore refer to a model prepared this way as **Ordinary Least Squares Linear Regression** or just **Least Squares Regression**."
43+
]
44+
},
45+
{
46+
"cell_type": "markdown",
47+
"metadata": {
48+
"id": "a7ibydXynwkK"
49+
},
50+
"source": [
51+
"The representation is a linear equation that combines a specific set of **input values (x)** the solution to which is the predicted output for that set of **output values (y)**. As such, both the input values (x) and the output value are numeric.\n",
52+
"\n",
53+
"The linear equation assigns one scale factor to each input value or column, called a **coefficient** and represented by the capital Greek letter **Beta (B)**. One additional coefficient is also added, giving the line an additional degree of freedom (e.g. moving up and down on a two-dimensional plot) and is often called the **intercept or the bias coefficient.**\n",
54+
"\n",
55+
"For example, in a simple regression problem (a single x and a single y), the form of the model would be:\n",
56+
"\n",
57+
"y = B0 + B1 * x1\n",
58+
"\n",
59+
"In higher dimensions when we have more than one input (x), the line is called a **plane or a hyper-plane**. The representation therefore is the form of the equation and the specific values used for the coefficients (e.g. B0 and B1 in the above example).\n",
60+
"\n",
61+
"It is common to talk about the complexity of a regression model like linear regression. This refers to the **number of coefficients used in the model.**\n",
62+
"\n",
63+
"When a **coefficient becomes zero,** it effectively removes the influence of the input variable on the model and therefore from the prediction made from the model (0 * x = 0). This becomes relevant if you look at **regularization methods** that change the learning algorithm to reduce the complexity of regression models by putting pressure on the absolute size of the coefficients, driving some to zero."
64+
]
65+
},
66+
{
67+
"cell_type": "markdown",
68+
"metadata": {
69+
"id": "R7lnhXAzoleK"
70+
},
71+
"source": [
72+
"**Simple Linear Regression**\n",
73+
"\n",
74+
"With simple linear regression when we have a *single input*, we can use statistics to estimate the coefficients\n",
75+
"\n",
76+
"**Ordinary Least Squares**\n",
77+
" - When we have *more than one input* we can use Ordinary Least Squares to estimate the values of the coefficients.\n",
78+
" - The *Ordinary Least Squares procedure seeks to minimize the sum of the squared residuals.* This means that given a regression line through the data we calculate the distance from each data point to the regression line, square it, and sum all of the squared errors together. This is the quantity that ordinary least squares seeks to minimize.\n",
79+
"\n",
80+
"**Regularization**\n",
81+
" - There are extensions of the training of the linear model called regularization methods. \n",
82+
" - These seek to *both minimize the sum of the squared error of the model on the training data (using ordinary least squares) but also to reduce the complexity of the model (like the number or absolute size of the sum of all coefficients in the model).*\n",
83+
" - Two popular examples of regularization procedures for linear regression are:\n",
84+
" 1. **Lasso Regression:** where Ordinary Least Squares is modified to also minimize the absolute sum of the coefficients (called L1 regularization).\n",
85+
" 2. **Ridge Regression:** where Ordinary Least Squares is modified to also minimize the squared absolute sum of the coefficients (called L2 regularization).\n",
86+
" - *These methods are effective to use when there is collinearity in your input values and ordinary least squares would overfit the training data.*\n",
87+
"\n",
88+
"\n",
89+
"**Data Preparation for Linear Regression Model**\n",
90+
"1. **Linear Assumption.** \n",
91+
" - Linear regression assumes that the relationship between your input and output is linear. It does not support anything else. \n",
92+
" - This may be obvious, but it is good to remember when you have a lot of attributes. You may need to transform data to make the relationship linear (e.g. log transform for an exponential relationship).\n",
93+
"2. **Remove Noise.**\n",
94+
" - Linear regression assumes that your input and output variables are not noisy. Consider using data cleaning operations that let you better expose and clarify the signal in your data. \n",
95+
" - This is most important for the output variable and you want to remove outliers in the output variable (y) if possible.\n",
96+
"3. **Remove Collinearity.** \n",
97+
" - Linear regression will over-fit your data when you have highly correlated input variables. Consider calculating pairwise correlations for your input data and removing the most correlated.\n",
98+
"4. **Gaussian Distributions.** \n",
99+
"- Linear regression will make more reliable predictions if your input and output variables have a Gaussian distribution. \n",
100+
"- You may get some benefit using transforms (e.g. log or BoxCox) on you variables to make their distribution more Gaussian looking.\n",
101+
"5. **Rescale Inputs:** \n",
102+
" - Linear regression will often make more reliable predictions if you rescale input variables using standardization or normalization."
103+
]
104+
},
105+
{
106+
"cell_type": "markdown",
107+
"metadata": {
108+
"id": "FUOE9uSRrEca"
109+
},
110+
"source": [
111+
"**Assumptions of Linear Regression**\n",
112+
"\n",
113+
"Linear regression is an analysis that assesses whether one or more predictor variables explain the dependent (criterion) variable. The regression has five key assumptions:\n",
114+
"1. Linear relationship\n",
115+
"2. Multivariate normality\n",
116+
"3. No or little multicollinearity\n",
117+
"4. No auto-correlation\n",
118+
"5. Homoscedasticity\n",
119+
"\n",
120+
"A note about sample size. In Linear regression the sample size rule of thumb is that the regression analysis requires at least 20 cases per independent variable in the analysis."
121+
]
122+
},
28123
{
29124
"cell_type": "code",
30125
"metadata": {
31126
"id": "VGe8PLvZqOpy",
32127
"colab": {
33128
"base_uri": "https://localhost:8080/"
34129
},
35-
"outputId": "fbd45f76-d99a-4595-9d33-f4f1b66bcacc"
130+
"outputId": "6e549da6-7c08-40df-8d75-5759a479ae7a"
36131
},
37132
"source": [
38133
"# Libraries \n",
@@ -68,7 +163,7 @@
68163
"base_uri": "https://localhost:8080/",
69164
"height": 204
70165
},
71-
"outputId": "9678cb52-bc86-4dab-b4f7-746dcb0074a6"
166+
"outputId": "d26013be-da7a-4fcb-9edc-b79aeeb7e58e"
72167
},
73168
"source": [
74169
"# Boston Dataset for Regression\n",
@@ -231,7 +326,7 @@
231326
"colab": {
232327
"base_uri": "https://localhost:8080/"
233328
},
234-
"outputId": "f29260b9-a025-41dc-c064-2159b10c02f0"
329+
"outputId": "baeb118a-b574-4953-8911-1149cb253f64"
235330
},
236331
"source": [
237332
"# Dataset overall Information\n",
@@ -275,7 +370,7 @@
275370
"colab": {
276371
"base_uri": "https://localhost:8080/"
277372
},
278-
"outputId": "88107e5a-48bc-4dbf-d7bd-76565c9507be"
373+
"outputId": "0034a431-4553-456d-f8c9-a9757d35fa19"
279374
},
280375
"source": [
281376
"# Let set the BASE MODEL on which we will improve\n",
@@ -312,7 +407,7 @@
312407
"base_uri": "https://localhost:8080/"
313408
},
314409
"id": "SrUy95nr3NOo",
315-
"outputId": "cf4d6d11-dc54-4d18-e303-5875b45141ee"
410+
"outputId": "51914598-0e9b-442b-ddc3-d8d59079aae2"
316411
},
317412
"source": [
318413
"# Add constant to the dataframe\n",
@@ -343,7 +438,7 @@
343438
"base_uri": "https://localhost:8080/",
344439
"height": 709
345440
},
346-
"outputId": "58797f03-78ba-4194-e02a-50193118017f"
441+
"outputId": "84174b49-11de-4af1-d88a-3a49a3efb08d"
347442
},
348443
"source": [
349444
"# Base Model\n",
@@ -371,10 +466,10 @@
371466
" <th>Method:</th> <td>Least Squares</td> <th> F-statistic: </th> <td> 89.01</td> \n",
372467
"</tr>\n",
373468
"<tr>\n",
374-
" <th>Date:</th> <td>Tue, 23 Nov 2021</td> <th> Prob (F-statistic):</th> <td>4.90e-115</td>\n",
469+
" <th>Date:</th> <td>Tue, 07 Dec 2021</td> <th> Prob (F-statistic):</th> <td>4.90e-115</td>\n",
375470
"</tr>\n",
376471
"<tr>\n",
377-
" <th>Time:</th> <td>14:19:50</td> <th> Log-Likelihood: </th> <td> -1548.6</td> \n",
472+
" <th>Time:</th> <td>06:55:54</td> <th> Log-Likelihood: </th> <td> -1548.6</td> \n",
378473
"</tr>\n",
379474
"<tr>\n",
380475
" <th>No. Observations:</th> <td> 506</td> <th> AIC: </th> <td> 3123.</td> \n",
@@ -456,8 +551,8 @@
456551
"Dep. Variable: HOUSEPRICE R-squared: 0.684\n",
457552
"Model: OLS Adj. R-squared: 0.677\n",
458553
"Method: Least Squares F-statistic: 89.01\n",
459-
"Date: Tue, 23 Nov 2021 Prob (F-statistic): 4.90e-115\n",
460-
"Time: 14:19:50 Log-Likelihood: -1548.6\n",
554+
"Date: Tue, 07 Dec 2021 Prob (F-statistic): 4.90e-115\n",
555+
"Time: 06:55:54 Log-Likelihood: -1548.6\n",
461556
"No. Observations: 506 AIC: 3123.\n",
462557
"Df Residuals: 493 BIC: 3178.\n",
463558
"Df Model: 12 \n",
@@ -506,39 +601,37 @@
506601
"**OUTPUT EXPLAINATION**\n",
507602
"\n",
508603
"**Omnibus/Prob(Omnibus)**\n",
509-
"- Omnibus describes the normalcy of the distribution of our residuals using skew and kurtosis as measurements. A 0 would indicate perfect normalcy. \n",
510-
"- Prob(Omnibus) is a statistical test measuring the probability the residuals are normally distributed. 1 would indicate perfectly normal distribution. \n",
511-
"- Skew is a measurement of symmetry in our data, with 0 being perfect symmetry. \n",
512-
"- Kurtosis measures the peakiness of our data, or its concentration around 0 in a normal curve. Higher kurtosis implies fewer outliers.\n",
513-
"- The Prob(Omnibus) indicates the probability that the residuals are normally distributed. \n",
514-
"- We hope to see something close to 1 here. \n",
604+
"- **Prob(Omnibus)** is a statistical test measuring the *probability the residuals are normally distributed. 1 would indicate perfectly normal distribution.* \n",
605+
"- **Omnibus** describes the normalcy of the *distribution of our residuals* using skew and kurtosis as measurements. A *0 would indicate perfect normalcy.* \n",
606+
"- The **Prob(Omnibus)** indicates the probability that the residuals are normally distributed. We hope to see something close to 1 here. \n",
515607
"- In this case Omnibus = 267 (way higher than 1) and Prob(Omnibus) = 0 which (normally = 1)is low. So the data is not normal, not ideal. \n",
516608
"\n",
517-
"**Skew -** \n",
518-
"- Measure of data symmetry.\n",
609+
"**Skew** \n",
610+
"- Skew is a measurement of symmetry in our data, with 0 being perfect symmetry. \n",
519611
"- We want to see something close to zero, indicating the residual distribution is normal. \n",
520612
"- Note that this value also drives the Omnibus\n",
521613
"- In this case, Skewness = 2.1, way higher than 0 so error/residual is skewwed\n",
522614
"\n",
523-
"**Kurtosis -** \n",
524-
"- Measure of \"peakiness\", or curvature of the data. \n",
525-
"- Kurtosis of the normal distribution is 3.0.\n",
615+
"**Kurtosis** \n",
616+
"- Kurtosis measures the *peakiness of our data.*\n",
617+
"- *Kurtosis of the normal distribution is 3.0*\n",
526618
"- In this case, Kurtosis = 13.14 which is way too higher\n",
527619
"\n",
528-
"**Durbin-Watson -**\n",
529-
"- Durbin-Watson is a measurement of homoscedasticity, or an even distribution of errors throughout our data. - Heteroscedasticity would imply an uneven distribution, for example as the data point grows higher the relative error grows higher. Ideal homoscedasticity will lie between 1 and 2. \n",
620+
"**Durbin-Watson**\n",
621+
"- Durbin-Watson is a *measurement of homoscedasticity, or an even distribution of errors throughout our data.* \n",
622+
"- Heteroscedasticity would imply an uneven distribution, for example as the data point grows higher the relative error grows higher. *Ideal homoscedasticity will lie between 1 and 2.* \n",
530623
"- In this case, Durbin-Watson = 0.78 is close, but within limits.\n",
531624
"\n",
532625
"**Jarque-Bera (JB)/Prob(JB) -** \n",
533-
"- Jarque-Bera (JB) and Prob(JB) are alternate methods of measuring the same value as Omnibus and Prob(Omnibus) using skewness and kurtosis. We use these values to confirm each other\n",
626+
"- Jarque-Bera (JB) and Prob(JB) are *alternate methods of measuring the same value as Omnibus and Prob(Omnibus)* using skewness and kurtosis. We use these values to confirm each other\n",
534627
"- It is like the Omnibus test in that it tests both skew and kurtosis. \n",
535628
"- It is also performed for the distribution analysis of the regression errors.\n",
536-
"- A large value of JB test indicates that the errors are not normally distributed.\n",
629+
"- A *large value of JB test indicates that the errors are not normally distributed.*\n",
537630
"- In this case, JB = 2542 which is way too higher so error are not normally distributed\n",
538631
"\n",
539632
"**Condition Number -**\n",
540633
"- Condition number is a measurement of the sensitivity of our model as compared to the size of changes in the data it is analyzing. \n",
541-
"- Multicollinearity is strongly implied by a high condition number. \n",
634+
"- *Multicollinearity is strongly implied by a high condition number.* \n",
542635
"- Multicollinearity a term to describe two or more independent variables that are strongly related to each other and are falsely affecting our predicted variable by redundancy.\n",
543636
"- When we have multicollinearity, we can expect much higher fluctuations to small changes in the data hence, we hope to see a relatively small number, something below 30. \n",
544637
"- In this case, Condition Number = well above 30, so multicollinearity present\n",
@@ -547,7 +640,7 @@
547640
"- R Squared is the measurement of how much of the independent variable is explained by changed in our dependent variable\n",
548641
"- In percentage terms, 0.68 would mean our model explains 68% of the dependent variable\n",
549642
"- Adjusted R-squared is important for analyzing multiple dependent variables’ efficacy on the model. \n",
550-
"- Linear regression has the quality that your model’s R-squared value will never go down with additional variables, only equal or higher. \n",
643+
"- R-squared value will never go down with additional variables, only equal or higher. \n",
551644
"- Therefore, your model could look more accurate with multiple variables even if they are poorly contributing. \n",
552645
"- The adjusted R-squared penalizes the R-squared formula based on the number of variables, therefore a lower adjusted score may be telling you some variables are not contributing to your model’s R-squared properly.\n",
553646
"- Both measures model performance and Possible values range from 0.0 to 1.0. \n",

0 commit comments

Comments
 (0)