Module2B-Workshop-part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Machine Learning: the Naive Bayes Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Naive Bayes is a simple technique for constructing classifiers: models that assign class labels to problem instances, represented as vectors of feature values. There is not a single algorithm for training such classifiers, but a family of algorithms based on a common principle: all naive Bayes classifiers assume that the value of a particular feature is independent of the value of any other feature, given the class variable. For instance, if we want to describe an apple, features such as  color or shape would be considered independent from the fruit and with different probability distributions. In this workshop, we will explore two major algorithms for training a Naive Bayes classifier: the Gaussian Naive Bayes and the MultinomialNaive Baayes (there are others, of course).\n",
    "\n",
    "Abstractly, naive Bayes is a conditional probability model: given a problem instance to be classified, represented by a vector $x =(x_{1},\\dots ,x_{n})$, representing some *N* features or pieces of evidence (independent variables), it assigns to this instance probabilities\n",
    "\n",
    "$Pr( C_k | x_1, x_2,..., x_N)$\n",
    "\n",
    "for each of K possible outcomes or classes, $C_K$.\n",
    "\n",
    "The problem with the above formulation is that if the number of features n is large or if a feature can take on a large number of values, then basing such a model on probability tables is infeasible. We therefore reformulate the model to make it more tractable. Using Bayes' theorem, the conditional probability can be decomposed as\n",
    "\n",
    "$ Pr( C | x_1, x_2, ..., x_N ) = \\frac{Pr(C) Pr(x_1, x_2, ..., x_N | C)}{Pr( x_1, x_2, ..., x_N)} $\n",
    "\n",
    "In plain English, using Bayesian probability terminology, the above equation can be written as\n",
    "\n",
    "$ posterior = \\frac{ prior x likelihood }{evidence} $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A General Machine Learning Architecture"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"images/ML.png\" width=\"700px\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Task A: A Breast Cancer Classifier Using Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this workshop, we will be applying Naive Bayes to try to predict if some tumor is malignant (cancer) or benign. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import the Required Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numerical Data Manipulation libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import statistics as stat\n",
    "\n",
    "# Figure Plotting libraries\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib.pyplot import figure\n",
    "import seaborn as sns\n",
    "sns.set()\n",
    "\n",
    "# Naive Bayes libraries\n",
    "import sklearn\n",
    "from sklearn.naive_bayes import BernoulliNB      # Naive Bayes Classifier based on a Bernoulli Distribution\n",
    "from sklearn.naive_bayes import GaussianNB       # Naive Bayes Classifier based on a Gaussian Distribution\n",
    "from sklearn.naive_bayes import MultinomialNB    # Naive Bayes Classifier based on a Multinomial Distribution\n",
    "\n",
    "# Machine Learning libraries\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import accuracy_score\n",
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "# Text Analysis libraries\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.pipeline import make_pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Some Functions\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to plot gaussian distribution in data\n",
    "def func_plot_gaussian( data, ylim, xlim ):\n",
    "   # xlim = (5, 40)\n",
    "   # ylim = (5, 30)\n",
    "\n",
    "    # separate the benign tumours (diagnosis = 0) from the malignant ones (diagnosis = 1)\n",
    "    mal = data[ data[ 'diagnosis'] == 1]\n",
    "    ben = data[ data[ 'diagnosis'] == 0]\n",
    "\n",
    "    # need to convert dataframe into a matrix in order to make the plot work\n",
    "    X = data[ ['radius_mean', 'texture_mean'] ]\n",
    "    x = X.to_numpy()\n",
    "\n",
    "    # plot figure\n",
    "    fig=plt.figure()\n",
    "    ax= fig.add_subplot(111)\n",
    "    \n",
    "    plt.scatter(mal['radius_mean'], mal['texture_mean'], c='r', marker='s', s=3, label='malignant')\n",
    "    plt.scatter(ben['radius_mean'], ben['texture_mean'], c='b', marker='o', s=3, label='benign')\n",
    "    plt.ylabel('texture_mean', fontsize=12)\n",
    "    plt.xlabel('radius_mean', fontsize=12)\n",
    "    plt.title('Breast Tumors', fontsize=14)\n",
    "    plt.legend()\n",
    "\n",
    "    xg = np.linspace(xlim[0], xlim[1], 60)\n",
    "    yg = np.linspace(ylim[0], ylim[1], 40)\n",
    "    xx, yy = np.meshgrid(xg, yg)\n",
    "    Xgrid = np.vstack([xx.ravel(), yy.ravel()]).T\n",
    "\n",
    "    for label, color in enumerate(['blue', 'red']):\n",
    "        mask = (y == label)\n",
    "        mu, std = x[mask].mean(0), x[mask].std(0)\n",
    "        P = np.exp(-0.5 * (Xgrid - mu) ** 2 / std ** 2).prod(1)\n",
    "        Pm = np.ma.masked_array(P, P < 0.05)\n",
    "        ax.pcolorfast(xg, yg, Pm.reshape(xx.shape), alpha=0.5, cmap=color.title() + 's')\n",
    "        ax.contour(xx, yy, P.reshape(xx.shape), levels=[0.01, 0.1, 0.5, 0.9], colors=color, alpha=0.2) \n",
    "    \n",
    "    ax.set(xlim=xlim, ylim=ylim)\n",
    "    fig.show()\n",
    "    return"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load breast cancer dataset\n",
    "# Data describes if a tumour is MALIGNANT (value 1) or BENIGN (value 0) accordong to:\n",
    "# - mean radius of the tumour\n",
    "# - mean texture of the tumour\n",
    "file_path = 'data/breast_data_simple.csv'\n",
    "data = pd.read_csv( file_path )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate your dataset: \n",
    "# put the variable that you wish to classify (or predict) in one variable\n",
    "# put your sources of evidence (or your features) in another variable\n",
    "y = data['diagnosis']                        # variable to classify and preduict\n",
    "X = data[['radius_mean', 'texture_mean']]    # variable containing your features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we proceed with any data analysis, we need to try to understand what kind of data are we dealing with. \n",
    "Naive Bayes model (like most machine learning models) are based on statistical analysis. This means that the distribution of your data plays an important role in how the machine learning algorithm should address it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for plotting purposes:\n",
    "# separate the benign tumors (diagnosis = 0) from the malignant ones (diagnosis = 1)\n",
    "malignant = data[ data[ 'diagnosis'] == 1]\n",
    "benign = data[ data[ 'diagnosis'] == 0]\n",
    "\n",
    "# need to convert dataframe into a matrix in order to make the plot work\n",
    "x = X.to_numpy()\n",
    "\n",
    "# plot figure\n",
    "fig=plt.figure()\n",
    "\n",
    "plt.scatter(malignant['radius_mean'], malignant['texture_mean'], c='r', marker='s', s=3, label='malignant')\n",
    "plt.scatter(benign['radius_mean'], benign['texture_mean'], c='b', marker='o', s=3, label='benign')\n",
    "plt.ylabel('texture_mean', fontsize=12)\n",
    "plt.xlabel('radius_mean', fontsize=12)\n",
    "plt.title('Breast Tumors', fontsize=14)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset is not sparse, which is good (it is hard to model sparse data). The data sems to be concentric and to deviate around a mean value. In statistics and machine learning, we usually represent data with a Gaussian Distribution (which is nothing more than a bell shaped curve). Let's see this in our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plotting function defined in the begining of the notebook\n",
    "func_plot_gaussian( data, (5, 42), (5, 30) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Running the Naive Bayes Classifier with a Gaussian Kernel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### The Importance of Defining Test and Training Sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Definition of the Type of Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GaussNB = GaussianNB()\n",
    "GaussNB.fit(X_train, y_train)\n",
    "print(GaussNB)\n",
    "\n",
    "y_prediction = GaussNB.predict(X_test)\n",
    "print( 'The overall accuracy of the model is ' + str(accuracy_score( y_test, y_prediction )))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Validating Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trials = 500\n",
    "\n",
    "accuracy = [None] * trials\n",
    "for trial in range( 0, trials ):\n",
    "    \n",
    "    # randomly select a test set and a training set\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3)\n",
    "    y_expected = y_test\n",
    "    \n",
    "    GaussNB = GaussianNB()                     # create the Gaussian Naive Bayes Classifer\n",
    "    GaussNB.fit(X_train, y_train)              # fit the model to the training data\n",
    "\n",
    "    y_predicted = GaussNB.predict(X_test)                       # get predictions of model on the test set\n",
    "    accuracy[trial] = accuracy_score( y_expected, y_predicted ) # save accuracy obtained in each trial\n",
    "    print(\"Applying Naive Bayes............................ Trial #\" + str(trial + 1) + \" ....... acc = \" + str( accuracy[trial] ))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Computute overall average accuracy\n",
    "avg_accuracy = stat.mean( accuracy )\n",
    "\n",
    "# plot results\n",
    "plt.figure()\n",
    "plt.scatter( range( 0, trials ), accuracy, s = 2 )\n",
    "lst = np.ones(trials, float)\n",
    "plt.plot( range( 0, trials ), avg_accuracy*lst, c='r' )\n",
    "plt.ylabel('Accuracy', fontsize=12)\n",
    "plt.xlabel('Number of Trials', fontsize=12)\n",
    "plt.title('Average Accuracy of a Naive Bayes Classifier using a Gaussian Kernel', fontsize=14)\n",
    "plt.show()\n",
    "\n",
    "print( \"Average model accuracy is \" +  str( avg_accuracy ) )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### What if we use a Bernoulli distribution?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "BernNB = BernoulliNB()\n",
    "BernNB.fit(X_train, y_train)\n",
    "print(BernNB)\n",
    "\n",
    "y_prediction = BernNB.predict(X_test)\n",
    "print( 'The overall accuracy of the model is ' + str(accuracy_score( y_test, y_prediction )))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happen to the performance of our classifier? How can you justify this?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Try it Yourself! Breast Cancer classification with more features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load your dataset\n",
    "file_path = 'data/breast_data_full.csv'\n",
    "data_full = pd.read_csv( file_path )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# what are the features in this dataset? How many are there?\n",
    "\n",
    "# YOUR CODE HERE:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate your dataset: \n",
    "# put the variable that you wish to classify (or predict) in one variable\n",
    "# put your sources of evidence (or your features) in another variable\n",
    "\n",
    "# note that your dataset contains a column id, which is not necessary. \n",
    "# hint: use Python's .loc method to try to sellect the columns that you want to use for your analysis\n",
    "\n",
    "# YOUR CODE HERE:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# separate the dataset into test set and train set\n",
    "\n",
    "# YOUR CODE HERE:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the NaiveBayes Gaussian kernel\n",
    "# Fit a model to the data\n",
    "# Use the learned model to try to predict the tumors on the testset\n",
    "# Measure the overall accuracy of the model\n",
    "\n",
    "\n",
    "# YOUR CODE HERE:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Comments\n",
    "What were your findings? Did the incorporation of more features have any impact on the predictions?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Naive Bayes in Text Classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "data = fetch_20newsgroups()\n",
    "data.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity here, we will select just a few of these categories, and download the training and testing set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = ['talk.religion.misc', 'soc.religion.christian', 'sci.space', 'comp.graphics']\n",
    "#categories = data.target_names\n",
    "\n",
    "train = fetch_20newsgroups(subset='train', categories=categories)\n",
    "test = fetch_20newsgroups(subset='test', categories=categories)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With any of the preceding examples, it can quickly become tedious to do the transformations by hand, especially if you wish to string together multiple steps. For example, we might want a processing pipeline that looks something like this:\n",
    "\n",
    "Impute missing values using the mean\n",
    "Transform features to quadratic\n",
    "Fit a linear regression\n",
    "To streamline this type of processing pipeline, Scikit-Learn provides a Pipeline object, which can be used as follows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define test and training sets\n",
    "X_train_raw = train.data\n",
    "y_train = train.target\n",
    "\n",
    "X_test_raw = test.data\n",
    "y_test = test.target\n",
    "y_expected = test.target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(X_train_raw[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract Tf.Idf features from text\n",
    "vec = TfidfVectorizer()\n",
    "X_train = vec.fit_transform( X_train_raw )\n",
    "X_test = vec.fit_transform( X_test_raw )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = make_pipeline(TfidfVectorizer(), MultinomialNB())\n",
    "\n",
    "model.fit(train.data, train.target)\n",
    "labels = model.predict(test.data)\n",
    "\n",
    "print( 'The overall accuracy of the model is ' + str(accuracy_score( y_expected, labels )))\n",
    "\n",
    "colormap = \"YlOrBr\" # more colors can be found here: https://matplotlib.org/tutorials/colors/colormaps.html\n",
    "mat = confusion_matrix(test.target, labels)\n",
    "sns.heatmap(mat.T, square=True, annot=True, fmt='d', cbar=True, cmap = colormap,\n",
    "            xticklabels=train.target_names, yticklabels=train.target_names)\n",
    "\n",
    "plt.xlabel('true label')\n",
    "plt.ylabel('predicted label');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### testing different sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_category(s, train=train, model=model):\n",
    "    pred = model.predict([s])\n",
    "    return train.target_names[pred[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_category('Teaching data analytics with really nice graphics')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_category('Flat earth people say Australia does not exist and we are all being paid by Nasa')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_category('Chef Kiko is opening a restaurant in Mars')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "predict_category('discussing islam vs atheism')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Raw Cell Format",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}