11.05.The_random_forest_exercise.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Ee-v8I_P8rfp"
   },
   "source": [
    "<div align=\"center\" style=\" font-size: 80%; text-align: center; margin: 0 auto\">\n",
    "<img src=\"https://raw.githubusercontent.com/Explore-AI/Pictures/master/Python-Notebook-Banners/Exercise.png\"  style=\"display: block; margin-left: auto; margin-right: auto;\";/>\n",
    "</div>\n",
    "\n",
    "# Exercise: The random forest\n",
    "© ExploreAI Academy\n",
    "\n",
    "In this exercise, we build, evaluate, and compare random forest regression models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ITqrvAqq8xSC"
   },
   "source": [
    "## Learning objectives\n",
    "\n",
    "By the end of this train, you should be able to:\n",
    "* Build a random forest regression model in Python.\n",
    "* Experiment with different numbers of trees.\n",
    "* Evaluate feature importance using a random forest. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercises\n",
    "\n",
    "In this exercise, we will be using the `Crop_yield` dataset which contains various factors that could influence the yield of a particular crop across different regions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import libraries and dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Region</th>\n",
       "      <th>Temperature</th>\n",
       "      <th>Rainfall</th>\n",
       "      <th>Soil_Type</th>\n",
       "      <th>Fertilizer_Usage</th>\n",
       "      <th>Pesticide_Usage</th>\n",
       "      <th>Irrigation</th>\n",
       "      <th>Crop_Variety</th>\n",
       "      <th>Yield</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>East</td>\n",
       "      <td>23.152156</td>\n",
       "      <td>803.362573</td>\n",
       "      <td>Clayey</td>\n",
       "      <td>204.792011</td>\n",
       "      <td>20.767590</td>\n",
       "      <td>1</td>\n",
       "      <td>Variety B</td>\n",
       "      <td>40.316318</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>West</td>\n",
       "      <td>19.382419</td>\n",
       "      <td>571.567670</td>\n",
       "      <td>Sandy</td>\n",
       "      <td>256.201737</td>\n",
       "      <td>49.290242</td>\n",
       "      <td>0</td>\n",
       "      <td>Variety A</td>\n",
       "      <td>26.846639</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>North</td>\n",
       "      <td>27.895890</td>\n",
       "      <td>-8.699637</td>\n",
       "      <td>Loamy</td>\n",
       "      <td>222.202626</td>\n",
       "      <td>25.316121</td>\n",
       "      <td>0</td>\n",
       "      <td>Variety C</td>\n",
       "      <td>-0.323558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>East</td>\n",
       "      <td>26.741361</td>\n",
       "      <td>897.426194</td>\n",
       "      <td>Loamy</td>\n",
       "      <td>187.984090</td>\n",
       "      <td>17.115362</td>\n",
       "      <td>0</td>\n",
       "      <td>Variety C</td>\n",
       "      <td>45.440871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>East</td>\n",
       "      <td>19.090286</td>\n",
       "      <td>649.384694</td>\n",
       "      <td>Loamy</td>\n",
       "      <td>110.459549</td>\n",
       "      <td>24.068804</td>\n",
       "      <td>1</td>\n",
       "      <td>Variety B</td>\n",
       "      <td>35.478118</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Region  Temperature    Rainfall Soil_Type  Fertilizer_Usage  \\\n",
       "0   East    23.152156  803.362573    Clayey        204.792011   \n",
       "1   West    19.382419  571.567670     Sandy        256.201737   \n",
       "2  North    27.895890   -8.699637     Loamy        222.202626   \n",
       "3   East    26.741361  897.426194     Loamy        187.984090   \n",
       "4   East    19.090286  649.384694     Loamy        110.459549   \n",
       "\n",
       "   Pesticide_Usage  Irrigation Crop_Variety      Yield  \n",
       "0        20.767590           1    Variety B  40.316318  \n",
       "1        49.290242           0    Variety A  26.846639  \n",
       "2        25.316121           0    Variety C  -0.323558  \n",
       "3        17.115362           0    Variety C  45.440871  \n",
       "4        24.068804           1    Variety B  35.478118  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load dataset\n",
    "df= pd.read_csv(\"https://raw.githubusercontent.com/Explore-AI/Public-Data/master/Data/Python/Crop_yield.csv\")\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing the dataset\n",
    "\n",
    "In the code below, we prepare our dataset for modelling by encoding categorical variables to convert them to a numeric format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Dummy Variable Encoding for categorical variables\n",
    "df_encoded = pd.get_dummies(df, drop_first=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1\n",
    "\n",
    "Create a function named `train_rf_model` to train and evaluate a random forest regression model on the encoded dataset. \n",
    "\n",
    "The function should take in three parameters:\n",
    "- A DataFrame containing the encoded features.\n",
    "- A string containing the name of the target variable.\n",
    "- The number of estimators for the random forest. \n",
    "\n",
    "It then returns: \n",
    "- The trained model object. \n",
    "- The RMSE and R<sup>2</sup> scores of the model's performance on the test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_rf_model(enc_df, target_var, n_est):\n",
    "  x = enc_df.drop(columns=target_var)\n",
    "  y = enc_df[target_var]\n",
    "  \n",
    "  x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=42)\n",
    "\n",
    "  model = RandomForestRegressor(n_estimators=n_est, random_state=42)\n",
    "  model.fit(x_train, y_train)\n",
    "\n",
    "  y_pred = model.predict(x_test)\n",
    "\n",
    "  r2 = metrics.r2_score(y_test, y_pred)\n",
    "  mse = metrics.mean_squared_error(y_test, y_pred)\n",
    "\n",
    "  return model, r2, mse"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2\n",
    "\n",
    "Use the function you have defined in **Exercise 1** to train and evaluate three different random forest regression models with each having the following number of estimators respectively: `50`, `100`, and `200`. Store the results in a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimators = [50, 100, 200]\n",
    "models = {}\n",
    "\n",
    "for i, est in enumerate(estimators):\n",
    "  model, r2, mse = train_rf_model(df_encoded, 'Yield', est)\n",
    "  models[f'model {i+1}'] = [model, r2, mse]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'model 1': [RandomForestRegressor(n_estimators=50, random_state=42),\n",
       "  0.9920180175887953,\n",
       "  0.739261264251345],\n",
       " 'model 2': [RandomForestRegressor(random_state=42),\n",
       "  0.9921300365756436,\n",
       "  0.7288864859605081],\n",
       " 'model 3': [RandomForestRegressor(n_estimators=200, random_state=42),\n",
       "  0.9922259008186051,\n",
       "  0.7200078994393476]}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3\n",
    "\n",
    "Say we wish to understand which features have the most impact on crop yield predictions.\n",
    "\n",
    "Use the `feature_importances_` attribute from our last trained random forest model in **Exercise 2** to return a series containing the feature importance score for each of the features in our dataset, sorted in descending order. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Rainfall                  0.978910\n",
       "Fertilizer_Usage          0.016670\n",
       "Temperature               0.001971\n",
       "Pesticide_Usage           0.001102\n",
       "Irrigation                0.000251\n",
       "Crop_Variety_Variety B    0.000202\n",
       "Region_West               0.000194\n",
       "Soil_Type_Loamy           0.000161\n",
       "Soil_Type_Sandy           0.000158\n",
       "Crop_Variety_Variety C    0.000143\n",
       "Region_North              0.000120\n",
       "Region_South              0.000118\n",
       "dtype: float64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index = models['model 3'][0].feature_names_in_\n",
    "values = models['model 3'][0].feature_importances_\n",
    "\n",
    "features_importance = pd.Series(values, index=index)\n",
    "features_importance.sort_values(ascending= False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Solutions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_rf_model(data, target_variable, n_estimators):\n",
    "\n",
    "    # Splitting the dataset into features and target variable\n",
    "    X = data.drop(target_variable, axis=1)  # Features\n",
    "    y = data[target_variable]  # Target variable\n",
    "\n",
    "    # Splitting the dataset into training and testing sets\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
    "\n",
    "    # Initializing the RandomForestRegressor with n_estimators\n",
    "    rf_model = RandomForestRegressor(n_estimators=n_estimators, random_state=42)\n",
    "\n",
    "    # Training the model on the training set\n",
    "    rf_model.fit(X_train, y_train)\n",
    "\n",
    "    # Making predictions on the test set\n",
    "    y_pred = rf_model.predict(X_test)\n",
    "\n",
    "    # Evaluating the model\n",
    "    mse = metrics.mean_squared_error(y_test, y_pred)  # Setting squared=False returns the RMSE\n",
    "    r2 = metrics.r2_score(y_test, y_pred)\n",
    "    \n",
    "    # Return the trained model and its performance metrics\n",
    "    return rf_model, {'MSE': mse, 'R2': r2}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function `train_rf_model` is designed to train and evaluate a random forest regression model. \n",
    "\n",
    "It takes three parameters: `data`, `target_variable`, and `n_estimators`.\n",
    "\n",
    "The function returns two items: the trained random forest model `rf_model` and a dictionary containing the evaluation metrics, `mse` and `r2`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'50 trees': {'MSE': 0.739261264251345, 'R2': 0.9920180175887953},\n",
       " '100 trees': {'MSE': 0.7288864859605081, 'R2': 0.9921300365756436},\n",
       " '200 trees': {'MSE': 0.7200078994393476, 'R2': 0.9922259008186051}}"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Number of estimators to evaluate\n",
    "estimators_list = [50, 100, 200]\n",
    "\n",
    "# Dictionary to store results\n",
    "results = {}\n",
    "\n",
    "# Train and evaluate models with different numbers of estimators\n",
    "for n in estimators_list:\n",
    "    # Store the entire returned dictionary as the value for each key\n",
    "    model, metric = train_rf_model(df_encoded, 'Yield', n)\n",
    "    results[f\"{n} trees\"] = metric\n",
    "    \n",
    "results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code above, we use the previously created function to train and evaluate multiple random forest models, each with a different number of trees (estimators). \n",
    "\n",
    "The for loop iterates over each value in `estimators_list`, where it calls the `train_rf_model()` function, passing the required parameters including the current number of estimators `n` as arguments.\n",
    "\n",
    "The two items returned by the function are stored in separate variables, `model` and `metric`.\n",
    "\n",
    "The `results` dictionary is then used to store the evaluation metrics for each model trained with a different number of trees. The keys are strings indicating the number of trees, and the values are the dictionary of metrics returned by the function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Rainfall                  0.978910\n",
       "Fertilizer_Usage          0.016670\n",
       "Temperature               0.001971\n",
       "Pesticide_Usage           0.001102\n",
       "Irrigation                0.000251\n",
       "Crop_Variety_Variety B    0.000202\n",
       "Region_West               0.000194\n",
       "Soil_Type_Loamy           0.000161\n",
       "Soil_Type_Sandy           0.000158\n",
       "Crop_Variety_Variety C    0.000143\n",
       "Region_North              0.000120\n",
       "Region_South              0.000118\n",
       "dtype: float64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Extract feature importances from the model\n",
    "feature_importances = model.feature_importances_\n",
    "\n",
    "# Get the names of the features, excluding the target variable 'Yield'\n",
    "feature_names =df_encoded.drop('Yield', axis=1).columns\n",
    "\n",
    "# Create a Pandas series \n",
    "importances = pd.Series(feature_importances, index=feature_names)\n",
    "\n",
    "# Sort the feature importances in descending order\n",
    "sorted_importances = importances.sort_values(ascending=False)\n",
    "sorted_importances"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the code above, we use the `feature_importances_` attribute of the trained random forest model to extract the importance scores for each feature. \n",
    "\n",
    "The variable `feature_names` stores the list of feature names that were used to train the model. This will be used for mapping each importance score to its corresponding feature name.\n",
    "\n",
    "`importances` is a Pandas series object where each feature's importance score is associated with its name. \n",
    "\n",
    "In `sorted_importances`, we get the importances sorted in descending order to get a quick view of the features considered most important by the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Which top two features contribute the most to the model's predictive ability?\n",
    "\n",
    "Understanding feature importance and the contribution of each variable to the model's predictions offers us an opportunity to streamline our models. This understanding enables us to focus on the most influential features, thereby reducing model complexity without significantly sacrificing performance.\n",
    "\n",
    "In refining your model, you should consider an experiment: retrain the model using only the subset of features that have demonstrated the highest importance scores. This encourages an exploration into how much we can reduce complexity while maintaining, or even potentially improving, model accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\" style=\" font-size: 80%; text-align: center; margin: 0 auto\">\n",
    "<img src=\"https://raw.githubusercontent.com/Explore-AI/Pictures/master/ExploreAI_logos/EAI_Blue_Dark.png\"  style=\"width:200px\";/>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyPZFCZhaikX+N2/Bg7W6SY+",
   "collapsed_sections": [],
   "name": "Search_algorithms.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "interpreter": {
   "hash": "6b5ebbc2c6bde2831bc6c0426f75aca8137ccfc69d329557556ed73faee126ae"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}