Step5_LightGBM Implementation and Optimization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Surprisingly, I have only heard about the Light Gradient Boosting Machine (LGBM) two days ago when I was working on a Machine Learning project with a friend. I immediately went online and did some research about it and found an article describing how the LGBM is the new XGBoost in the sense that this is now the ultimate weapon for Data Scientists on Kaggle competitions as well as in the industry. As I have mentioned previously, the next stage is to implement LSTM Neural Network on the stock dataset which would conclude the project. But I felt that it was worth my time trying out the LGBM considering that I have been dealing with much larger datasets lately for both Kaggle competitions and my other Data Science projects.\n",
    "\n",
    "The main advantage of the LGBM to XGBoost is its significantly faster processing speed in model building and this was found to be true. However, the LGBM creators also highlighted LGBM's improved accuracy over XGBoost but in this case this was not found to hold true. Given that the average speed to do GridSearch with 5 parameters (each with 3 to 5 possible values) only took about a minute, and the resulting performance here (accuracy and F1-score) was found to be only marginally inferior to the XGBoost, it was found that the LGBM model to be a great alternative to XGBoost when time is a limiting factor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\" First import all the required libraries \"\"\"\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import f1_score\n",
    "import lightgbm as lgb\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import TimeSeriesSplit\n",
    "from sklearn.metrics import fbeta_score, make_scorer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\" Import the required datasets \"\"\"\n",
    "SP500 = pd.read_csv('SP500_2.csv', parse_dates=True)\n",
    "Nasdaq = pd.read_csv('Nasdaq_2.csv', parse_dates=True)\n",
    "DJI = pd.read_csv('DJI_2.csv', parse_dates=True)\n",
    "DAX = pd.read_csv('DAX_2.csv', parse_dates=True)\n",
    "Paris = pd.read_csv('Paris_2.csv', parse_dates=True)\n",
    "Tokyo = pd.read_csv('Tokyo_2.csv', parse_dates=True)\n",
    "HongKong = pd.read_csv('HongKong_2.csv', parse_dates=True)\n",
    "Aus = pd.read_csv('Aus_2.csv', parse_dates=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Since we had previously exported our engineered features as a CSV file,\n",
    "we can just import it in instead of going through the feature engineering\n",
    "procedure with our own functions again\n",
    "\"\"\"\n",
    "reduced_megaset = pd.read_csv('reduced_megaset.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\" The target variable is the daily return of each day, binary encoded \"\"\"\n",
    "target_raw = (SP500['Adj Close'].shift(-1)/SP500['Adj Close'])-1\n",
    "target = target_raw[21:]\n",
    "target[target > 0] = 1\n",
    "target[target <= 0] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "Split our dimension_reduced megaset and the target array into training and testing\n",
    "subsets\n",
    "\"\"\"\n",
    "X_train = reduced_megaset[:6001]\n",
    "X_test = reduced_megaset[6001:-1]\n",
    "y_train = target[:6001]\n",
    "y_test = target[6001:-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LightGBM Classifier [Accuracy score: 0.5783, f1-score: 0.7155]\n"
     ]
    }
   ],
   "source": [
    "\"\"\" LightGBM Implementation and Results\"\"\"\n",
    "# We need to convert our training data into LightGBM dataset format\n",
    "d_train = lgb.Dataset(X_train, label=y_train)\n",
    "# Setting parameters for training\n",
    "# objective set to binary for binary classification problem\n",
    "# boosting_type set to gbdt for gradient boosting\n",
    "# binary_logloss as metric for binary classification predictions\n",
    "# other parameters randomly selected and subject to change for optimization\n",
    "params = {'boosting_type': 'gbdt',\n",
    "          'learning_rate': 0.003,\n",
    "          'max_depth': 10,\n",
    "          'metric': 'binary_logloss',\n",
    "          'min_data': 50,\n",
    "          'num_leaves': 10,\n",
    "          'objective': 'binary',\n",
    "          'sub_feature': 0.5}\n",
    "# fit the clf_LGBM on training data with 100 training iterations\n",
    "clf_LGBM = lgb.train(params, d_train, 100)\n",
    "# make predictions with test data\n",
    "y_pred = clf_LGBM.predict(X_test)\n",
    "# sinec the output is a list of probabilities, below we have converted the probabilities\n",
    "# to binary prediction with threshold set at 0.5\n",
    "for i in range(0, len(y_pred)):\n",
    "    if y_pred[i] >= 0.5:\n",
    "       y_pred[i] = 1\n",
    "    else:  \n",
    "       y_pred[i]=0\n",
    "# evaluate predictions with accuracy metric\n",
    "clf_LGBM_accuracy = accuracy_score(y_test, y_pred)\n",
    "# evaluate predictions with F1-score metric\n",
    "clf_LGBM_f1 = f1_score(y_test, y_pred)\n",
    "print(\"LightGBM Classifier [Accuracy score: {:.4f}, f1-score: {:.4f}]\".format(clf_LGBM_accuracy, clf_LGBM_f1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is a demonstration of how the LGBM hyperparameters were fine-tuned using GridSearch() with cross-validation using TimeSeriesSplit(). Please note that it took many attempts adjusting the range of values for each parameter to get to what is shown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 324 candidates, totalling 972 fits\n",
      "LGBM Classifier Optimized [Accuracy score: 0.5739, f1-score: 0.7263]\n",
      "{'colsample_bytree': 0.58, 'learning_rate': 0.001, 'n_estimators': 75, 'num_leaves': 17, 'subsample': 0.5}\n",
      "0.532\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done 972 out of 972 | elapsed:  1.3min finished\n"
     ]
    }
   ],
   "source": [
    "\"\"\" Optimization of LightGBM \"\"\"\n",
    "# Choose LGBM Classifier as the algorithm for optimization with GridSearch\n",
    "clf_LGBM2 = lgb.LGBMClassifier(boosting_type = 'gbdt', metric = 'binary_logloss', \n",
    "                               min_data = 50, objective = 'binary', sub_feature = 0.5)\n",
    "# Create a dictionary for the parameters\n",
    "gridParams = {'learning_rate': [0.0001, 0.0003, 0.0005, 0.001],'n_estimators': [75, 100, 125],\n",
    "             'num_leaves': [15, 16, 17],'colsample_bytree' : [0.58, 0.60, 0.62],'subsample' : [0.4, 0.5, 0.7]}\n",
    "# Choose the time series cross-validator\n",
    "tscv = TimeSeriesSplit(n_splits=3)\n",
    "# Create the GridSearch object\n",
    "grid = GridSearchCV(clf_LGBM2, gridParams, verbose=1, cv= tscv)\n",
    "# Fit the grid search object to the data to compute the optimal model\n",
    "grid_fit_LGBM = grid.fit(X_train, y_train)\n",
    "# Return the optimal model after fitting the data\n",
    "best_clf_LGBM = grid_fit_LGBM.best_estimator_\n",
    "# Make predictions with the optimal model\n",
    "best_predictions_LGBM = best_clf_LGBM.predict(X_test)\n",
    "# Get the accuracy and F1_score of the optimized model\n",
    "clf_LGBM_optimized_accuracy = accuracy_score(y_test, best_predictions_LGBM)\n",
    "clf_LGBM_optimized_f1 = f1_score(y_test, best_predictions_LGBM)\n",
    "print(\"LGBM Classifier Optimized [Accuracy score: {:.4f}, f1-score: {:.4f}]\".format(clf_LGBM_optimized_accuracy, clf_LGBM_optimized_f1))\n",
    "print(grid_fit_LGBM.best_params_)\n",
    "print(grid_fit_LGBM.best_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our previous results (Step4):\n",
    "XGB Classifier: [Accuracy: 0.5435, F1-score: 0.6809] Improved XGB Classifier: [Accuracy: 0.5739, f1-score: 0.7168] XGB Classifier Optimized [Accuracy score: 0.5826, f1-score: 0.7363]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}