ML0120EN-1.4-Review-LogisticRegressionwithTensorFlow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://www.bigdatauniversity.com\"><img src=\"https://ibm.box.com/shared/static/qo20b88v1hbjztubt06609ovs85q8fau.png\" width=\"400px\"  align=\"center\"></a>\n",
    "<h1 align=\"center\"><font size=\"5\">LOGISTIC REGRESSION WITH TENSORFLOW</font></h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "\n",
    "Logistic Regression is one of most important techniques in data science. It is usually used to solve the classic classification problem.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
    "<font size = 3><strong>This lesson covers the following concepts of Logistics Regression:</strong></font>\n",
    "<br>\n",
    "<h2>Table of Contents</h2>\n",
    "\n",
    "<ol>\n",
    "    <li><a href=\"#ref1\">Linear Regression vs Logistic Regression</a></li>\n",
    "    <li><a href=\"#ref2\">Utilizing Logistic Regression in TensorFlow</a></li>\n",
    "    <li><a href=\"#ref3\">Training</a></li>\n",
    "</ol>    \n",
    "</div>\n",
    "<p></p>\n",
    "<br>\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref1\"></a>\n",
    "<h2>What is different between Linear and Logistic Regression?</h2>\n",
    "\n",
    "While Linear Regression is suited for estimating continuous values (e.g. estimating house price), it is n0t the best tool for predicting the class in which an observed data point belongs. In order to provide estimate for classification, we need some sort of guidance on what would be the <b>most probable class</b> for that data point. For this, we use <b>Logistic Regression</b>.\n",
    "\n",
    "<div class=\"alert alert-success alertsuccess\" style=\"margin-top: 20px\">\n",
    "<font size=\"3\"><strong>Recall linear regression:</strong></font>\n",
    "<br>\n",
    "<br>\n",
    "Linear regression finds a function that relates a continuous dependent variable, <i>y</i>, to some predictors (independent variables <i>x1</i>, <i>x2</i>, etc.). Simple linear regression assumes a function of the form:\n",
    "<br><br>\n",
    "$$\n",
    "y = w0 + w1 \\times x1 + w2 \\times x2 + \\cdots\n",
    "$$\n",
    "<br>\n",
    "and finds the values of <i>w0</i>, <i>w1</i>, <i>w2</i>, etc. The term <i>w0</i> is the \"intercept\" or \"constant term\" (it's shown as <i>b</i> in the formula below):\n",
    "<br><br>\n",
    "$$\n",
    "Y = W X + b\n",
    "$$\n",
    "<p></p>\n",
    "\n",
    "</div>\n",
    "\n",
    "Logistic Regression is a variation of Linear Regression, useful when the observed dependent variable, <i>y</i>, is categorical. It produces a formula that predicts the probability of the class label as a function of the independent variables.\n",
    "\n",
    "Despite the name logistic <i>regression</i>, it is actually a <b>probabilistic classification</b> model. Logistic regression fits a special s-shaped curve by taking the linear regression and transforming the numeric estimate into a probability with the following function:\n",
    "\n",
    "$$\n",
    "ProbabilityOfaClass = \\theta(y) = \\frac{e^y}{1 + e^y} = exp(y) / (1 + exp(y)) = p \n",
    "$$\n",
    "\n",
    "which produces p-values between 0 (as y approaches minus infinity $-\\infty$) and 1 (as y approaches plus infinity $+\\infty$). This now becomes a special kind of non-linear regression.\n",
    "\n",
    "In this equation, <i>y</i> is the regression result (the sum of the variables weighted by the coefficients), <code>exp</code> is the exponential function and $\\theta(y)$ is the <a href=\"http://en.wikipedia.org/wiki/Logistic_function\">logistic function</a>, also called logistic curve. It is a common \"S\" shape (sigmoid curve), and was first developed for modeling population growth.\n",
    "\n",
    "You might also have seen this function before, in another configuration:\n",
    "\n",
    "$$\n",
    "ProbabilityOfaClass = \\theta(y) = \\frac{1}{1+e^{-y}}\n",
    "$$\n",
    "\n",
    "So, briefly, Logistic Regression passes the input through the logistic/sigmoid function but then treats the result as a probability:\n",
    "\n",
    "<img src=\"https://ibm.box.com/shared/static/kgv9alcghmjcv97op4d6onkyxevk23b1.png\" width=\"400\" align=\"center\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref2\"></a>\n",
    "<h2>Utilizing Logistic Regression in TensorFlow</h2>\n",
    "\n",
    "For us to utilize Logistic Regression in TensorFlow, we first need to import the required libraries. To do so, you can run the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import time\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection import train_test_split\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will load the dataset we are going to use. In this case, we are utilizing the <code>iris</code> dataset, which is inbuilt -- so there's no need to do any preprocessing and we can jump right into manipulating it. We separate the dataset into <i>xs</i> and <i>ys</i>, and then into training <i>xs</i> and <i>ys</i> and testing <i>xs</i> and <i>ys</i>, (pseudo)randomly."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Understanding the Data</h3>\n",
    "\n",
    "<h4><code>Iris Dataset</code>:</h4>\n",
    "This dataset was introduced by British Statistician and Biologist Ronald Fisher, it consists of 50 samples from each of three species of Iris (Iris setosa, Iris virginica and Iris versicolor). In total it has 150 records under five attributes - petal length, petal width, sepal length, sepal width and species. <a href=\"https://archive.ics.uci.edu/ml/datasets/iris\">Dataset source</a>\n",
    "\n",
    "Attributes\n",
    "Independent Variable\n",
    "<ul>\n",
    "    <li>petal length</li>\n",
    "    <li>petal width</li>\n",
    "    <li>sepal length</li>\n",
    "    <li>sepal width</li>\n",
    "</ul>\n",
    "Dependent Variable\n",
    "<ul>    \n",
    "    <li>Species\n",
    "        <ul>\n",
    "            <li>Iris setosa</li>\n",
    "            <li>Iris virginica</li>\n",
    "            <li>Iris versicolor</li>\n",
    "        </ul>\n",
    "    </li>\n",
    "</ul>\n",
    "<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "iris = load_iris()\n",
    "iris_X, iris_y = iris.data[:-1,:], iris.target[:-1]\n",
    "iris_y= pd.get_dummies(iris_y).values\n",
    "trainX, testX, trainY, testY = train_test_split(iris_X, iris_y, test_size=0.33, random_state=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we define x and y. These placeholders will hold our iris data (both the features and label matrices), and help pass them along to different parts of the algorithm. You can consider placeholders as empty shells into which we insert our data. We also need to give them shapes which correspond to the shape of our data. Later, we will insert data into these placeholders by “feeding” the placeholders the data via a “feed_dict” (Feed Dictionary).\n",
    "\n",
    "<h3>Why use Placeholders?</h3>\n",
    "\n",
    "<ol>\n",
    "    <li>This feature of TensorFlow allows us to create an algorithm which accepts data and knows something about the shape of the data without knowing the amount of data going in.</li>\n",
    "    <li>When we insert “batches” of data in training, we can easily adjust how many examples we train on in a single step without changing the entire algorithm.</li>\n",
    "</ol>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# numFeatures is the number of features in our input data.\n",
    "# In the iris dataset, this number is '4'.\n",
    "numFeatures = trainX.shape[1]\n",
    "\n",
    "# numLabels is the number of classes our data points can be in.\n",
    "# In the iris dataset, this number is '3'.\n",
    "numLabels = trainY.shape[1]\n",
    "\n",
    "\n",
    "# Placeholders\n",
    "# 'None' means TensorFlow shouldn't expect a fixed number in that dimension\n",
    "X = tf.placeholder(tf.float32, [None, numFeatures]) # Iris has 4 features, so X is a tensor to hold our data.\n",
    "yGold = tf.placeholder(tf.float32, [None, numLabels]) # This will be our correct answers matrix for 3 classes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Set model weights and bias</h3>\n",
    "\n",
    "Much like Linear Regression, we need a shared variable weight matrix for Logistic Regression. We initialize both <code>W</code> and <code>b</code> as tensors full of zeros. Since we are going to learn <code>W</code> and <code>b</code>, their initial value does not matter too much. These variables are the objects which define the structure of our regression model, and we can save them after they have been trained so we can reuse them later.\n",
    "\n",
    "We define two TensorFlow variables as our parameters. These variables will hold the weights and biases of our logistic regression and they will be continually updated during training. \n",
    "\n",
    "Notice that <code>W</code> has a shape of [4, 3] because we want to multiply the 4-dimensional input vectors by it to produce 3-dimensional vectors of evidence for the difference classes. <code>b</code> has a shape of [3] so we can add it to the output. Moreover, unlike our placeholders above which are essentially empty shells waiting to be fed data, TensorFlow variables need to be initialized with values, e.g. with zeros."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "W = tf.Variable(tf.zeros([4, 3]))  # 4-dimensional input and  3 classes\n",
    "b = tf.Variable(tf.zeros([3])) # 3-dimensional output [0,0,1],[0,1,0],[1,0,0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Randomly sample from a normal distribution with standard deviation .01\n",
    "\n",
    "weights = tf.Variable(tf.random_normal([numFeatures,numLabels],\n",
    "                                       mean=0,\n",
    "                                       stddev=0.01,\n",
    "                                       name=\"weights\"))\n",
    "\n",
    "bias = tf.Variable(tf.random_normal([1,numLabels],\n",
    "                                    mean=0,\n",
    "                                    stddev=0.01,\n",
    "                                    name=\"bias\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h3>Logistic Regression model</h3>\n",
    "\n",
    "We now define our operations in order to properly run the Logistic Regression. Logistic regression is typically thought of as a single equation:\n",
    "\n",
    "$$\n",
    "ŷ =sigmoid(WX+b)\n",
    "$$\n",
    "\n",
    "However, for the sake of clarity, we can have it broken into its three main components: \n",
    "- a weight times features matrix multiplication operation, \n",
    "- a summation of the weighted features and a bias term, \n",
    "- and finally the application of a sigmoid function. \n",
    "\n",
    "As such, you will find these components defined as three separate operations below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Three-component breakdown of the Logistic Regression equation.\n",
    "# Note that these feed into each other.\n",
    "apply_weights_OP = tf.matmul(X, weights, name=\"apply_weights\")\n",
    "add_bias_OP = tf.add(apply_weights_OP, bias, name=\"add_bias\") \n",
    "activation_OP = tf.nn.sigmoid(add_bias_OP, name=\"activation\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we have seen before, the function we are going to use is the <i>logistic function</i> $(\\frac{1}{1+e^{-Wx}})$, which is fed the input data after applying weights and bias. In TensorFlow, this function is implemented as the <code>nn.sigmoid</code> function. Effectively, this fits the weighted input with bias into a 0-100 percent curve, which is the probability function we want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref3\"></a>\n",
    "<h2>Training</h2>\n",
    "\n",
    "The learning algorithm is how we search for the best weight vector (${\\bf w}$). This search is an optimization problem looking for the hypothesis that optimizes an error/cost measure.\n",
    "\n",
    "<b>What tell us our model is bad?</b>  \n",
    "The Cost or Loss of the model, so what we want is to minimize that. \n",
    "\n",
    "<b>What is the cost function in our model?</b>  \n",
    "The cost function we are going to utilize is the Squared Mean Error loss function.\n",
    "\n",
    "<b>How to minimize the cost function?</b>   \n",
    "We can't use <b>least-squares linear regression</b> here, so we will use <a href=\"http://en.wikipedia.org/wiki/Gradient_descent\">gradient descent</a> instead. Specifically, we will use batch gradient descent which calculates the gradient from all data points in the data set.\n",
    "\n",
    "<h3>Cost function</h3>\n",
    "Before defining our cost function, we need to define how long we are going to train and how should we define the learning rate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Number of Epochs in our training\n",
    "numEpochs = 700\n",
    "\n",
    "# Defining our learning rate iterations (decay)\n",
    "learningRate = tf.train.exponential_decay(learning_rate=0.0008,\n",
    "                                          global_step= 1,\n",
    "                                          decay_steps=trainX.shape[0],\n",
    "                                          decay_rate= 0.95,\n",
    "                                          staircase=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Defining our cost function - Squared Mean Error\n",
    "cost_OP = tf.nn.l2_loss(activation_OP-yGold, name=\"squared_error_cost\")\n",
    "\n",
    "#Defining our Gradient Descent\n",
    "training_OP = tf.train.GradientDescentOptimizer(learningRate).minimize(cost_OP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we move on to actually running our operations. We will start with the operations involved in the prediction phase (i.e. the logistic regression itself).\n",
    "\n",
    "First, we need to initialize our weights and biases with zeros or random values via the inbuilt Initialization Op, <b>tf.initialize_all_variables()</b>. This Initialization Op will become a node in our computational graph, and when we put the graph into a session, then the Op will run and create the variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Create a tensorflow session\n",
    "sess = tf.Session()\n",
    "\n",
    "# Initialize our weights and biases variables.\n",
    "init_OP = tf.global_variables_initializer()\n",
    "\n",
    "# Initialize all tensorflow variables\n",
    "sess.run(init_OP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want some additional operations to keep track of our model's efficiency over time. We can do this like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#VERY IMPORTANT TO UNDERSTAND BELOW STATEMENT!\n",
    "# argmax(activation_OP, 1) returns the label with the most probability\n",
    "# argmax(yGold, 1) is the correct label\n",
    "correct_predictions_OP = tf.equal(tf.argmax(activation_OP,1),tf.argmax(yGold,1))\n",
    "\n",
    "# If every false prediction is 0 and every true prediction is 1, the average returns us the accuracy\n",
    "accuracy_OP = tf.reduce_mean(tf.cast(correct_predictions_OP, \"float\"))\n",
    "\n",
    "# Summary op for regression output\n",
    "activation_summary_OP = tf.summary.histogram(\"output\", activation_OP)\n",
    "\n",
    "# Summary op for accuracy\n",
    "accuracy_summary_OP = tf.summary.scalar(\"accuracy\", accuracy_OP)\n",
    "\n",
    "# Summary op for cost\n",
    "cost_summary_OP = tf.summary.scalar(\"cost\", cost_OP)\n",
    "\n",
    "# Summary ops to check how variables (W, b) are updating after each iteration\n",
    "weightSummary = tf.summary.histogram(\"weights\", weights.eval(session=sess))\n",
    "biasSummary = tf.summary.histogram(\"biases\", bias.eval(session=sess))\n",
    "\n",
    "# Merge all summaries\n",
    "merged = tf.summary.merge([activation_summary_OP, accuracy_summary_OP, cost_summary_OP, weightSummary, biasSummary])\n",
    "\n",
    "# Summary writer\n",
    "writer = tf.summary.FileWriter(\"summary_logs\", sess.graph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can define and run the actual training loop, like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step 0, training accuracy 0.333333, cost 34.5208, change in cost 34.5208\n",
      "step 10, training accuracy 0.555556, cost 30.1848, change in cost 4.33607\n",
      "step 20, training accuracy 0.646465, cost 28.2457, change in cost 1.93904\n",
      "step 30, training accuracy 0.646465, cost 26.6016, change in cost 1.64416\n",
      "step 40, training accuracy 0.646465, cost 25.2307, change in cost 1.37082\n",
      "step 50, training accuracy 0.646465, cost 24.0957, change in cost 1.13507\n",
      "step 60, training accuracy 0.646465, cost 23.1548, change in cost 0.940866\n",
      "step 70, training accuracy 0.646465, cost 22.3703, change in cost 0.784472\n",
      "step 80, training accuracy 0.646465, cost 21.7107, change in cost 0.659616\n",
      "step 90, training accuracy 0.656566, cost 21.1507, change in cost 0.559994\n",
      "step 100, training accuracy 0.666667, cost 20.6705, change in cost 0.480167\n",
      "step 110, training accuracy 0.666667, cost 20.2548, change in cost 0.415771\n",
      "step 120, training accuracy 0.666667, cost 19.8914, change in cost 0.363407\n",
      "step 130, training accuracy 0.666667, cost 19.5709, change in cost 0.320454\n",
      "step 140, training accuracy 0.666667, cost 19.286, change in cost 0.284922\n",
      "step 150, training accuracy 0.676768, cost 19.0307, change in cost 0.255274\n",
      "step 160, training accuracy 0.686869, cost 18.8004, change in cost 0.230326\n",
      "step 170, training accuracy 0.686869, cost 18.5912, change in cost 0.209183\n",
      "step 180, training accuracy 0.69697, cost 18.4001, change in cost 0.191128\n",
      "step 190, training accuracy 0.707071, cost 18.2245, change in cost 0.175587\n",
      "step 200, training accuracy 0.717172, cost 18.0624, change in cost 0.162138\n",
      "step 210, training accuracy 0.737374, cost 17.9119, change in cost 0.150421\n",
      "step 220, training accuracy 0.737374, cost 17.7718, change in cost 0.14015\n",
      "step 230, training accuracy 0.747475, cost 17.6407, change in cost 0.1311\n",
      "step 240, training accuracy 0.757576, cost 17.5176, change in cost 0.123083\n",
      "step 250, training accuracy 0.777778, cost 17.4017, change in cost 0.115946\n",
      "step 260, training accuracy 0.787879, cost 17.2921, change in cost 0.10956\n",
      "step 270, training accuracy 0.787879, cost 17.1883, change in cost 0.10383\n",
      "step 280, training accuracy 0.787879, cost 17.0896, change in cost 0.0986481\n",
      "step 290, training accuracy 0.787879, cost 16.9957, change in cost 0.0939674\n",
      "step 300, training accuracy 0.787879, cost 16.906, change in cost 0.0897026\n",
      "step 310, training accuracy 0.79798, cost 16.8201, change in cost 0.0858173\n",
      "step 320, training accuracy 0.79798, cost 16.7379, change in cost 0.0822563\n",
      "step 330, training accuracy 0.79798, cost 16.6589, change in cost 0.0789852\n",
      "step 340, training accuracy 0.818182, cost 16.5829, change in cost 0.0759773\n",
      "step 350, training accuracy 0.818182, cost 16.5097, change in cost 0.0731926\n",
      "step 360, training accuracy 0.828283, cost 16.4391, change in cost 0.0706158\n",
      "step 370, training accuracy 0.838384, cost 16.3709, change in cost 0.0682163\n",
      "step 380, training accuracy 0.838384, cost 16.3049, change in cost 0.0659866\n",
      "step 390, training accuracy 0.838384, cost 16.241, change in cost 0.0639019\n",
      "step 400, training accuracy 0.848485, cost 16.179, change in cost 0.0619545\n",
      "step 410, training accuracy 0.848485, cost 16.1189, change in cost 0.0601254\n",
      "step 420, training accuracy 0.848485, cost 16.0605, change in cost 0.0584087\n",
      "step 430, training accuracy 0.858586, cost 16.0037, change in cost 0.0567894\n",
      "step 440, training accuracy 0.868687, cost 15.9485, change in cost 0.0552654\n",
      "step 450, training accuracy 0.868687, cost 15.8946, change in cost 0.0538206\n",
      "step 460, training accuracy 0.878788, cost 15.8422, change in cost 0.0524569\n",
      "step 470, training accuracy 0.878788, cost 15.791, change in cost 0.0511618\n",
      "step 480, training accuracy 0.878788, cost 15.7411, change in cost 0.0499315\n",
      "step 490, training accuracy 0.878788, cost 15.6923, change in cost 0.0487633\n",
      "step 500, training accuracy 0.878788, cost 15.6447, change in cost 0.0476456\n",
      "step 510, training accuracy 0.878788, cost 15.5981, change in cost 0.0465841\n",
      "step 520, training accuracy 0.888889, cost 15.5525, change in cost 0.0455704\n",
      "step 530, training accuracy 0.888889, cost 15.5079, change in cost 0.0445986\n",
      "step 540, training accuracy 0.89899, cost 15.4643, change in cost 0.0436697\n",
      "step 550, training accuracy 0.89899, cost 15.4215, change in cost 0.042778\n",
      "step 560, training accuracy 0.89899, cost 15.3796, change in cost 0.0419245\n",
      "step 570, training accuracy 0.89899, cost 15.3385, change in cost 0.0411034\n",
      "step 580, training accuracy 0.89899, cost 15.2981, change in cost 0.0403137\n",
      "step 590, training accuracy 0.89899, cost 15.2586, change in cost 0.0395517\n",
      "step 600, training accuracy 0.909091, cost 15.2198, change in cost 0.0388212\n",
      "step 610, training accuracy 0.909091, cost 15.1816, change in cost 0.0381155\n",
      "step 620, training accuracy 0.909091, cost 15.1442, change in cost 0.0374365\n",
      "step 630, training accuracy 0.909091, cost 15.1074, change in cost 0.0367765\n",
      "step 640, training accuracy 0.909091, cost 15.0713, change in cost 0.0361423\n",
      "step 650, training accuracy 0.909091, cost 15.0358, change in cost 0.0355272\n",
      "step 660, training accuracy 0.909091, cost 15.0008, change in cost 0.0349321\n",
      "step 670, training accuracy 0.909091, cost 14.9665, change in cost 0.0343552\n",
      "step 680, training accuracy 0.909091, cost 14.9327, change in cost 0.0337992\n",
      "step 690, training accuracy 0.909091, cost 14.8994, change in cost 0.0332556\n",
      "final accuracy on test set: 0.9\n"
     ]
    }
   ],
   "source": [
    "# Initialize reporting variables\n",
    "cost = 0\n",
    "diff = 1\n",
    "epoch_values = []\n",
    "accuracy_values = []\n",
    "cost_values = []\n",
    "\n",
    "# Training epochs\n",
    "for i in range(numEpochs):\n",
    "    if i > 1 and diff < .0001:\n",
    "        print(\"change in cost %g; convergence.\"%diff)\n",
    "        break\n",
    "    else:\n",
    "        # Run training step\n",
    "        step = sess.run(training_OP, feed_dict={X: trainX, yGold: trainY})\n",
    "        # Report occasional stats\n",
    "        if i % 10 == 0:\n",
    "            # Add epoch to epoch_values\n",
    "            epoch_values.append(i)\n",
    "            # Generate accuracy stats on test data\n",
    "            train_accuracy, newCost = sess.run([accuracy_OP, cost_OP], feed_dict={X: trainX, yGold: trainY})\n",
    "            # Add accuracy to live graphing variable\n",
    "            accuracy_values.append(train_accuracy)\n",
    "            # Add cost to live graphing variable\n",
    "            cost_values.append(newCost)\n",
    "            # Re-assign values for variables\n",
    "            diff = abs(newCost - cost)\n",
    "            cost = newCost\n",
    "\n",
    "            #generate print statements\n",
    "            print(\"step %d, training accuracy %g, cost %g, change in cost %g\"%(i, train_accuracy, newCost, diff))\n",
    "\n",
    "\n",
    "# How well do we perform on held-out test data?\n",
    "print(\"final accuracy on test set: %s\" %str(sess.run(accuracy_OP, \n",
    "                                                     feed_dict={X: testX, \n",
    "                                                                yGold: testY})))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Why don't we plot the cost to see how it behaves?</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jupyterlab/conda/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2957: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/home/jupyterlab/conda/lib/python3.6/site-packages/numpy/core/_methods.py:80: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xd8VfX9x/HXJ4tASEIgA0KYsocghCHLAQLFioi7aqmI1Fmt9def/tqfbe3WWmutC/dE1IJKwSIqiiwljLA3gYRAEobskPX9/ZHb3wMxgZDc5Nzc+34+HveRe0/OyX1zPb5z8j3LnHOIiEjoCPM6gIiI1C0Vv4hIiFHxi4iEGBW/iEiIUfGLiIQYFb+ISIhR8YuIhBgVv4hIiFHxi4iEmAivA1QkMTHRtW3b1usYIiL1xrJly/Y655KqMm9AFn/btm3JyMjwOoaISL1hZjuqOq+GekREQoyKX0QkxKj4RURCjIpfRCTEnLH4zewlM8s3szUnTetlZovNbLWZzTSzuEqWHW1mG81si5k94M/gIiJSPVXZ4n8FGH3KtBeAB5xzPYEZwH+dupCZhQNPAd8DugHXm1m3GqUVEZEaO2PxO+fmA/tPmdwZmO97Phe4soJF+wNbnHPbnHNFwNvA5TXIKiIiflDdMf41wFjf86uBVhXM0xLIPul1jm9arSguLePZL7aybMeB2noLEZGgUN3inwjcaWbLgFigqIJ5rIJpld7g18wmm1mGmWUUFBScdaCikjJeXZTFL99fQ0lp2VkvLyISKqpV/M65Dc65kc65vsBUYGsFs+Xw7b8E0oDc0/zMKc65dOdcelJSlc46/paYBhE89P1urN99iNeXVPkENhGRkFOt4jezZN/XMOCXwLMVzLYU6Ghm7cwsCrgO+LC6QatidI/mXNApicc+3kT+ocLafCsRkXqrKodzTgUWA53NLMfMbqH8CJ1NwAbKt+Jf9s2bamazAZxzJcBdwBxgPfCOc25t7fwz/j8rvxnbnaLSMn43a31tvpWISL1lzlU67O6Z9PR0V5OLtD0+dxNPfLqZtyYNYFCHRD8mExEJTGa2zDmXXpV5g/LM3dsvPIc2zRrxyw/WUFSiHb0iIicLyuKPjgzn12O7s63gKM9/uc3rOCIiASUoix/gos7JjO7enCc/20z2/mNexxERCRhBW/wAD13WjTAzHv7XOq+jiIgEjKAu/tQmDblneEfmrsvjk3V5XscREQkIQV38ABOHtKNjcmN+PXMtx4tKvY4jIuK5oC/+yPAwfjuuBzkHjvP051u8jiMi4rmgL36Age2bMf68ljz3xTa2FhzxOo6IiKdCovgBHhzTlQaRYfzqg7UE4klrIiJ1JWSKPym2Af81qjMLtuxl1urdXscREfFMyBQ/wA0D2tCjZRwPz1zH4cJir+OIiHgipIo/PMz43bieFBw5wd8+2ex1HBERT4RU8QP0btWE6/u35pVFWazffcjrOCIidS7kih/g56M6E98wkl++v4ayMu3oFZHQEpLF36RRFA98rwvLdhzgveU5XscREalTIVn8AFf1SSO9TQJ/+mgD3xyr6JbBIiLBKWSLPyzM+O24Hhw8XswjczZ6HUdEpM6EbPEDdG0Rx48GtWXq1ztZsfOA13FEROpESBc/wL0jOpIc24D//WANpdrRKyIhIOSLPzY6kv/9fjfW7DrEm1/t8DqOiEitC/niB7i0ZwuGdEjk0TkbyT9c6HUcEZFapeIHzIyHL+/OieIy/mf6ah3bLyJBTcXv0z6pMf8zpgufrM/nqXm6br+IBC8V/0kmDGrLuN6p/PWTTXy+Md/rOCIitULFfxIz44/jz6VL8zjueXslO/cd8zqSiIjfqfhP0TAqnGdv7INzjtveWKb79IpI0Dlj8ZvZS2aWb2ZrTprW28yWmNlKM8sws/6VLFvqm2elmX3oz+C1qU2zGJ647jzW7znEL2as1h27RCSoVGWL/xVg9CnTHgF+45zrDTzke12R48653r7H2OrHrHsXdUnm3uGdmL5iF68v0fH9IhI8zlj8zrn5wP5TJwNxvufxQK6fcwWEuy/uwPAuyTw8cx0ZWad+BCIi9VN1x/jvBR41s2zgL8CDlcwX7RsKWmJm46r5Xp4JCzP+em1vWiY05I43l+vkLhEJCtUt/tuBnzrnWgE/BV6sZL7Wzrl04AfA38zsnMp+oJlN9v2SyCgoKKhmLP+LbxjJczf15XBhCXe+uZzi0jKvI4mI1Eh1i38CMN33/F2gwp27zrlc39dtwOfAeZX9QOfcFOdcunMuPSkpqZqxakeX5nH86cqeLM06wO9nrfc6johIjVS3+HOBC3zPLwa+c+dyM0swswa+54nAYGBdNd/Pc5f3bsnEwe14ZVEW76/Y5XUcEZFqizjTDGY2FbgQSDSzHOBXwK3AE2YWARQCk33zpgO3OecmAV2B58ysjPJfMH9yztXb4gd4cEwX1uQe5IHpq+iUEku31LgzLyQiEmAsEI9RT09PdxkZGV7HqFD+4UIue3IBDSLCmXnXEOIbRXodSUQEM1vm26d6Rjpz9ywlx0bz9A192X3wOPdOW6EreYpIvaPir4a+bRJ46LLuzNtYwBOffmf3hohIQFPxV9ONA1pzZZ80nvh0M5+uz/M6johIlan4q8nM+P0VPeieGse901aStfeo15FERKpExV8D0ZHhPHtjX8LDjB+/voxjRSVeRxIROSMVfw21atqIv193HpvyD/PAP3UlTxEJfCp+PxjWKYn7R3bmw8xcnv9ym9dxRERO64wncEnV3H7BOazLPcQfZm8gOTaacee19DqSiEiFVPx+EhZmPHZNL/YeOcH972bSNCaKYZ0C65pDIiKgoR6/io4M5/kJ6XRMieW2N5aRmf2N15FERL5Dxe9ncdGRvHpzP5rGRHHzK0vZrsM8RSTAqPhrQXJcNK9NLL9S9Q9f+or8Q7qBi4gEDhV/LWmf1JiXf9SPfUeKmPDyUg4VFnsdSUQEUPHXql6tmvDMjX3ZnHeYya9lUFhc6nUkEREVf227oFMSf7m6F0u27ee+d1ZSqqt5iojHdDhnHRh3Xkv2HjnB72atJ7HxWn4ztjtm5nUsEQlRKv46Mmloe/IPn2DK/G0kNW7A3cM7eh1JREKUir8OPTC6C3sPn+CxuZtIim3Adf1bex1JREKQir8OhYUZf77qXPYfK+J/ZqymaUwUI7s39zqWiIQY7dytY5HhYTx9Qx96pjXh7qkrWJq13+tIIhJiVPweaBQVwcs/6kfLhIbc8spSNu457HUkEQkhKn6PNI2J4rWJ/YmODGfCS1+z65vjXkcSkRCh4vdQWkIjXp3Yn6NFJfzwxa84cLTI60giEgJU/B7r2iKOF36YTvaB40x8dalu3ygitU7FHwAGtG/G36/rTWb2N0x+bRnHi3RpBxGpPSr+ADG6RwsevaoXi7buZcLLX3PkhLb8RaR2VKn4zewlM8s3szUnTettZkvMbKWZZZhZ/0qWnWBmm32PCf4KHoyu7JvGE9edx7IdB7jpxa84eFxX9BQR/6vqFv8rwOhTpj0C/MY51xt4yPf6W8ysKfArYADQH/iVmSVUO20IuKxXKk/9oA9rdh3khheWaIeviPhdlYrfOTcfOPVMIwfE+Z7HA7kVLDoKmOuc2++cOwDM5bu/QOQUo3s0Z8pN6WzKO8L1zy+h4PAJryOJSBCpyRj/vcCjZpYN/AV4sIJ5WgLZJ73O8U2TM7ioSzIvTehH1r6jXDdlMXm6i5eI+ElNiv924KfOuVbAT4EXK5inomsPV3hBejOb7NtXkFFQUFCDWMFjSMdEXps4gD0HC7nmucU6yUtE/KImxT8BmO57/i7lY/inygFanfQ6jYqHhHDOTXHOpTvn0pOSkmoQK7j0b9eU1ycNYP/RIq55djE79unm7SJSMzUp/lzgAt/zi4HNFcwzBxhpZgm+nbojfdPkLPRpncDUWwdyrKiEa55bzNaCI15HEpF6rKqHc04FFgOdzSzHzG4BbgUeM7NM4A/AZN+86Wb2AoBzbj/wW2Cp7/Gwb5qcpR4t45k6eSClZY5rn1uiC7uJSLWZc4F3D9j09HSXkZHhdYyAtCX/CDe8sISikjJev2UAPVrGex1JRAKAmS1zzqVXZV6duVvPdEhuzDs/Pp9GURFc//wSVuw84HUkEalnVPz1UJtmMUz78UCaxkRx4wtf8fV2jZ6JSNWp+OuptIRGTJt8Ps3jo5nw0tcs3LLX60giUk+o+Oux5vHRvD35fNo0a8TNryxl3sZ8ryOJSD2g4q/nkmIbMPXWgXRKaczk1zL4aPVuryOJSIBT8QeBhJgo3pw0kJ4t47njreU898VWAvFoLREJDCr+IBHfMJK3bh3ImJ4t+ONHG3hw+mqKS8u8jiUiASjC6wDiP9GR4Tx53Xm0T4zhyc+2sHP/MZ65oS/xjSK9jiYiAURb/EEmLMz42cjO/OXqXizN2s8VzyzU9X1E5FtU/EHqqr5pvH5L+cXdrnh6ERlZOtZfRMqp+IPYwPbNmHHHYOIbRvKD57/i/RW7vI4kIgFAxR/k2iXGMOOOQZzXugn3TlvJ43M36YgfkRCn4g8BTRpF8fotA7iqbxpPfLqZe95eSWFxqdexRMQjOqonRERFhPHoVefSLjGGR+dsZNc3x5lyU1+aNW7gdTQRqWPa4g8hZsadF3XgqR/0Yc2ug4x7eiGb83Rdf5FQo+IPQZee24K3Jw/keFEZ459ZxILNusCbSChR8Yeo81on8P6dg0iNb8iEl79m6tc7vY4kInVExR/C0hIa8d7t5zOkQyIPTl/NH2avp7RMR/yIBDsVf4iLjY7kxQnp3DSwDVPmb2PSq0s5cLTI61giUotU/EJEeBgPX96d317enQVb9nLp379kuW7pKBK0VPwClB/xc9P5bXnvtkGEhRnXPLuYF77cppO9RIKQil++pVerJsy6eygXdUnmd7PWc9sbyzh4vNjrWCLiRyp++Y74RpFMuakvv7y0K5+uz+f7T37J6pyDXscSET9R8UuFzIxJQ9sz7cfnU1rquPKZRby2OEtDPyJBQMUvp9W3TQKzfjKUwR2a8dAHa7lr6goOF2roR6Q+U/HLGSXERPHihH789+gu/HvNHsb+YyHrcg95HUtEqumMxW9mL5lZvpmtOWnaNDNb6XtkmdnKSpbNMrPVvvky/Blc6lZYmHH7hecw9daBHCsq4YqnFzL1650a+hGph6qyxf8KMPrkCc65a51zvZ1zvYF/AtNPs/xFvnnTqx9TAkX/dk2Z9ZOh9G/XlAenr+a+dzI5eqLE61gichbOWPzOuflAhfftMzMDrgGm+jmXBLDExg145eb+3HdJJ95fuYux/1jAJl3lU6TeqOkY/1Agzzm3uZLvO+BjM1tmZpNP94PMbLKZZZhZRkFBQQ1jSW0LDzN+Mrwjb94ygIPHSxj7jwW8tyzH61giUgU1Lf7rOf3W/mDnXB/ge8CdZjasshmdc1Occ+nOufSkpKQaxpK6MqhDIrPvGULvVk24/91M7pu2Uid8iQS4ahe/mUUA44Fplc3jnMv1fc0HZgD9q/t+EriSY6N5c9JA7hnekQ8ycxn1+Hy+2KS/2kQCVU22+EcAG5xzFf59b2YxZhb7n+fASGBNRfNK/RceZvz0kk7MuGMQjaMjmPDS1zw4fTVHtONXJOBU5XDOqcBioLOZ5ZjZLb5vXccpwzxmlmpms30vU4AFZpYJfA3Mcs7923/RJRCdm9aEf909hB8Pa8/bS3cy+m/zWbx1n9exROQkFojHYaenp7uMDB32X99lZO3n/nczydp3jJsHt+Xno7rQMCrc61giQcnMllX1sHmduSu1Jr1tU2bfM5QfDWrLywuzGPP3L1m2Q9f5F/Gail9qVaOoCH49tjtvTRpAUUkZVz+7iD99tIETJaVeRxMJWSp+qRODOiTy73uHck16K579YiuXPblAl3oW8YiKX+pMbHQkf7ryXF6+uR8Hjxcz7umFPD53E8WlZV5HEwkpKn6pcxd1Tubjey9gbK9Unvh0M+OeWsiGPbrap0hdUfGLJ+IbRfL4tb159sa+7DlYyNgnF/L051so0da/SK1T8YunRvdozsc/Hcbwrsk88u+NjH9mEatyvvE6lkhQU/GL55o1bsDTN/ThyevPY/fBQi5/aiEPfbCGQ7rTl0itUPFLQDAzLuuVyqc/u4AfDmzDG0t2cPFfvuCDlbt0sxcRP1PxS0CJi47kN5f34IM7h5DaJJp73l7JjS9+xdaCI15HEwkaKn4JSD3T4plxx2B+O64Hq3IO8r2/fcljH2+ksFgnfonUlIpfAlZ4mHHTwDZ89rMLufTcFjz52RYuefwL5m3I9zqaSL2m4peAlxTbgMev7c1btw4gKjyMm19Zym2vLyP3m+NeRxOpl1T8Um8MOieRj+4Zxn+N6sy8jfmM+OsXPD9/m878FTlLKn6pV6Iiwrjzog58ct8FDGzfjN/PXs9lTy4gI2u/19FE6g0Vv9RLrZo24sUJ6Tx3U18OHS/mqmcX8/P3Mtl/tMjraCIBT8Uv9ZaZMap7c+bedwE/Htae6ct3ccGj85gyf6su+yxyGip+qfdiGkTw4JiuzL5nKH1aJ/CH2RsY8dcvmLVqt07+EqmAil+CRqeUWF6d2J/XJvYnJiqCO99azlXPLmb5Tt31S+RkKn4JOsM6JTHrJ0P50/ie7Nx/jPFPL+Kut5aTvf+Y19FEAoJuti5B7eiJEp77YitTvtxGWRncPLgtd1zUgfiGkV5HE/Er3WxdxCemQQT3jezMvPsv5LJeqUz5chsXPjqPVxdl6fh/CVkqfgkJLeIb8tg1vZh51xC6NI/jVx+uZdTf5jN3XZ52AEvIUfFLSOnRMp63bh3ACz8s/4v41tcyuP75JazZpRu/S+hQ8UvIMTNGdEthzr3DePjy7mzKO8Jl/1jAfe+s1PV/JCRo566EvEOFxTw1bwsvL8wC4IYBrbn9wnNIjo32NpjIWfDrzl0ze8nM8s1szUnTppnZSt8jy8xWVrLsaDPbaGZbzOyBqv8TROpOXHQkD36vK5/97AKu6N2S1xbvYNgj8/jj7PW6BIQEpTNu8ZvZMOAI8JpzrkcF338MOOice/iU6eHAJuASIAdYClzvnFt3plDa4hcvZe09yhOfbub9lbtoFBnOxCHtmDS0vQ4BlYDm1y1+59x8oMJLH5qZAdcAUyv4dn9gi3Num3OuCHgbuLwqoUS81DYxhsev7c3H9w7jws7JPPnZFob++TOe/HQzR06UeB1PpMZqunN3KJDnnNtcwfdaAtknvc7xTauQmU02swwzyygoKKhhLJGa65gSy1M39GH2T4YyoH0zHpu7iaF//oznvtjK8SJdBE7qr5oW//VUvLUPYBVMq3RcyTk3xTmX7pxLT0pKqmEsEf/plhrH8z9M54M7B3NuWhP++NEGhj4yj5cWbNc9gKVeqnbxm1kEMB6YVsksOUCrk16nAbnVfT8Rr/Vq1YRXJ/bnvdvOp2NyYx7+1zoufPRz3liyg6ISnQUs9UdNtvhHABucczmVfH8p0NHM2plZFHAd8GEN3k8kIKS3bcrUyQN5a9IAWiY05Jfvr+Hixz7nnYxsSnQZCKkHqnI451RgMdDZzHLM7Bbft67jlGEeM0s1s9kAzrkS4C5gDrAeeMc5t9af4UW8NKhDIu/ddj6v3NyPpjFR/Py9VQz/6xdM/XqnbgQjAU0ncIn4gXOOuevyeGreFjJzDpIS14BJQ9rzgwGtiWkQ4XU8CQFnczinil/Ej5xzLNyyj6c/38Kirfto0iiSCee35UeD2pIQE+V1PAliKn6RALBi5wGe/nwrc9fl0SgqnBsGtGbS0PakxOlSEOJ/Kn6RALJxz2Ge+XwLM1ftJtyMK/umcdsF7WnTLMbraBJEVPwiAWjnvmM8N38r7y7LoaS0jO+fm8rtF55D1xZxXkeTIKDiFwlg+YcKeXHBdt5YsoOjRaUM75LMHRd1oG+bBK+jST2m4hepBw4eK+bVxVm8vHA7B44VM6BdU26/8Bwu6JRE+WWwRKpOxS9SjxwrKmHq19k8P38bew4V0iG5MRMHt2N8n5ZER4Z7HU/qCRW/SD1UVFLGrNW5vPDldtbmHiKhUSQ3DmzDTQPbkKwjgeQMVPwi9Zhzjq+27+fFBdv5ZH0eEWHGZb1SuWVIO7qnxnsdTwLU2RS/TikUCTBmxsD2zRjYvhlZe4/y8sLtvLssh+nLdzGwfVMmDWnPxV2SCQvTfgCpHm3xi9QDB48V8/bSnby6KIvcg4W0S4zh5sFtuapvGo2itP0mGuoRCVrFpWX8e80eXliwnczsb4iLjuD6Aa350aC2tIhv6HU88ZCKXyTIOedYvvMALy7Yzr/X7CHMjDE9W3Dz4Lb0btVEh4OGII3xiwQ5M6Nvm6b0bdOU7P3HeHVRFm8vzebDzFx6toznpoFtuKxXKg2jdDiofJe2+EWCxOHCYt5fsYvXl+xgU94R4qIjuDq9FTcMaE37pMZex5NapqEekRDmnOPr7ft546udfLR6NyVljiEdErlxYBtGdE0mIrymt9qWQKTiFxEA8g8X8s7SbN76aie5BwtpHhfNDwa05rp+rXRSWJBR8YvIt5SUlvHZhnxeX7KDLzfvJSLMGNW9OTcObMPA9k21MzgIaOeuiHxLRHgYI7s3Z2T35mzfe5Q3l+zg3WU5zFq9mw7JjblpYBuu6NOSuOhIr6NKHdAWv0iIKiwuZWZmLm8s2UFmzkEaRYVzee9Uru3Xml5p8foroJ7RUI+InJXM7G94Y8kOZq7KpbC4jM4psVzbrxVXnNdS9wquJ1T8IlIthwqLmZmZyztLs8nMOUhUeBgju6dwbb9WDD4nUdcHCmAqfhGpsXW5h3gnI5sZK3Zx8HgxaQkNubpvK65OTyO1iS4PEWhU/CLiN4XFpcxZu4d3MrJZuGUfZjCsYxLX9WvF8K4pREXovIBAoOIXkVqxc98x3l2WzbsZOew5VEizmCjG92nJtf1a0SE51ut4IU3FLyK1qrTMMX9TAW8v3cmn6/MpKXP0bZPANelpjOnZglgdFlrn/Fr8ZvYS8H0g3znX46TpdwN3ASXALOfczytYNgs4DJQCJVUNpeIXqT8KDp9g+vIcpmVks63gKA0iwhjVvTnj+7RkSIdEXSKijvi7+IcBR4DX/lP8ZnYR8AvgUufcCTNLds7lV7BsFpDunNt7Nv8AFb9I/eOcY0X2N0xfnsPMzN0cPF5MUmwDxvVOZXyfNLq2iPM6YlDz+1CPmbUF/nVS8b8DTHHOfXKG5bJQ8YuEnBMlpczbkM8/l+9i3obyoaCuLeK4sk9LxvZOJTlW1wnyt7oo/pXAB8BooBC43zm3tILltgMHAAc855ybcpr3mAxMBmjdunXfHTt2VCW/iAS4/UeLmJmZy/TlOWTmHCTMYFinJMb3SWNktxSiI3XPAH+oi+JfA3wG3AP0A6YB7d0pP8zMUp1zuWaWDMwF7nbOzT/T+2mLXyQ4bck/zPTlu5ixYhe7DxYS2yCCMT1bcGXfNNLbJOgEsRqoi4u05QDTfUX/tZmVAYlAwckzOedyfV/zzWwG0B84Y/GLSHDqkBzLz0d34f6RnVmybR//XL6LmatymZaRTVpCQ644ryVje6XSMUWHhtam6hb/+8DFwOdm1gmIAr41jm9mMUCYc+6w7/lI4OGahBWR4BAWZgzqkMigDon8dlx35qzdw/Tlu3hq3hae/GwLXZrHMrZ3Kpedm0qrpo28jht0qnJUz1TgQsq36POAXwGvAy8BvYEiysf4PzOzVOAF59wYM2sPzPD9mAjgLefc76sSSkM9IqEp/3Ahs1bt5sPMXFbs/AaAvm0SGNsrlTE9W5AU28DjhIFLJ3CJSL2Xvf8YH2bmMjMzlw17DhNmMLhDIpf1SmVU9+bEN9RJYidT8YtIUNm45zAfZu5iZuZudu4/RlR4GBd2TmJs71SGd0mhYZSODFLxi0hQcs6xMvsbPszM5V+rdlNw+AQxUeFc0i2Fsb1TGdoxicgQPVNYxS8iQa+0zPHVtn18mJnLR2v2cPB4MfENIxnZLYUx57Zg8DmJIXXlUBW/iISUopIy5m8qYNbq3XyyLo/DJ0qIi45gZPfmXNqzBYM7BP8vAd1sXURCSlREGCO6pTCiWwonSkr5ctNeZq/ezZw1e3hvWQ6x0RFc0i2FS3u2YEjHRBpEhPY+ARW/iASVBhHh3/olsHDLXmat2sPcdeXnCsQ2iGBEtxTG9GzB0I6JIXnJCA31iEhIKCopY+HWvcxetZuP1+Vx8HgxjRtEMKJrMmN6tmBYp6R6/UtAY/wiIqdRXFrGoq37mL1qN3PW7eGbY8XERIVzcdcURnVP4cLOyTRuUL8GRFT8IiJVVFxaxpJt+5i9ejcfr81j39EioiLCGNIhkVHdUxjRNYVmjQP/jGEVv4hINZSWOZbtOMCctXuYs3YPOQeOE2aQ3rYpI7ulMKp784C9dpCKX0SkhpxzrNt9iDlr8/h47R427DkMQLcWcYzq3pxRPVLonBKLWWBcSlrFLyLiZzv2HeXjtXnMWbuHZTsP4By0adbo//8S6NPa2/sJqPhFRGpR/uFCPlmXz5y1e1i0dS/FpY7Exg24pFsyI7qmMLhD3R8mquIXEakjhwuLmbexgDlr9/DFxgKOnCghOrJ85/CIrilc3CWZ5Ljav8ewztwVEakjsdGRjO2VytheqRSVlPHV9n18si6PT9bn88n6fAB6pcUzomsKw7um0LWF9/sFtMUvIlILnHNszDv8/78EVmaX31imZZOGDO+azPCuKQxs39Rvl4/QUI+ISIDJP1zIvA3lfwV8ubmAwuIyYqLCGdYpieFdU7ioc1KNzhdQ8YuIBLDC4lIWbd3LJ+vz+XR9HnmHTmAG/do25a1JA4ioxj0FNMYvIhLAoiPDubhLChd3ScGN68GaXYf4ZH0eeYcKq1X6Z0vFLyLiITOjZ1o8PdPi6+w9g/vOBCIi8h0qfhGREKPiFxEJMSp+EZEQo+IXEQkxKn4RkRCj4hcRCTEqfhGREBOQl2wwswJgRzUXTwT2+jGOvylfzShfzShfzQRyvjbOuaSqzBiQxV8TZpZR1eszWdU8AAAEF0lEQVRVeEH5akb5akb5aibQ81WVhnpEREKMil9EJMQEY/FP8TrAGShfzShfzShfzQR6vioJujF+ERE5vWDc4hcRkdOoV8VvZllmttrMVppZhm9aUzOba2abfV8TKll2gm+ezWY2oY6yPWpmG8xslZnNMLMmVV22NlSS8ddmtss3baWZjalk2dFmttHMtpjZA3WYb9pJ2bLMbGVVl62FfE3M7D3ff9P1ZnZ+oKx/p8kXMOtgJfkCaf2rKF/ArH9+5ZyrNw8gC0g8ZdojwAO+5w8Af65guabANt/XBN/zhDrINhKI8D3/c0XZKlu2Dj+/XwP3n2G5cGAr0B6IAjKBbnWR75TvPwY85NVnCLwKTPI9jwKaBMr6d5p8AbMOVpIvkNa/7+QLpPXPn496tcVficsp/w+G7+u4CuYZBcx1zu13zh0A5gKjazuYc+5j51yJ7+USIK2237OW9Ae2OOe2OeeKgLcp/9zrjJkZcA0wtS7f96T3jwOGAS8COOeKnHPfECDrX2X5AmUdPM3nVxW1vv6dKZ/X65+/1bfid8DHZrbMzCb7pqU453YD+L4mV7BcSyD7pNc5vmm1ne1kE4GPqrmsv1T2Pnf5hgJeqmSooi4+v9PlAxgK5DnnNldjWX9oDxQAL5vZCjN7wcxiCJz1r7J8J/NyHTxdvkBY/870+Xm9/vlVfSv+wc65PsD3gDvNbFgVl7MKpvn7cKZKs5nZL4AS4M2zXbYOMj4DnAP0BnZT/ufsqeri86ss339cz+m3tmr7M4wA+gDPOOfOA45SPrRTFXXx+Z02XwCsg5XlC5T170z/fb1e//yqXhW/cy7X9zUfmEH5n4B5ZtYCwPc1v4JFc4BWJ71OA3LrIBu+HXnfB25wvsHAqi7rbxW9j3MuzzlX6pwrA56v5L1r/fOrLB+AmUUA44FpZ7usH+UAOc65r3yv36O8KAJi/TtNvkBZByvMF0Dr3+k+v0BY//yq3hS/mcWYWex/nlO+02oN8CHwn6MkJgAfVLD4HGCkmSX4/pQc6ZtWq9nMbDTw38BY59yxs/x3+dVpMrY4abYrKnnvpUBHM2tnZlHAdZR/7rWez/ftEcAG51xONZb1C+fcHiDbzDr7Jg0H1hEA69/p8gXKOniafAGx/p3mvy8EwPrnd17vXa7qg/IxuEzfYy3wC9/0ZsCnwGbf16a+6enACyctPxHY4nvcXEfZtlA+NrnS93jWNz0VmH26Zevw83sdWA2sovx/phanZvS9HgNsovzoCr9nPN3nALwC3HbK/F58hr2BDN9n9T7lR+h4vv6dIV8grYMV5QuI9a+yfIG0/vnzoTN3RURCTL0Z6hEREf9Q8YuIhBgVv4hIiFHxi4iEGBW/iEiIUfGLiIQYFb+ISIhR8YuIhJj/Ax+cUaoM47chAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "plt.plot([np.mean(cost_values[i-50:i]) for i in range(len(cost_values))])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming no parameters were changed, you should reach a peak accuracy of 90% at the end of training, which is commendable. Try changing the parameters such as the length of training, and maybe some operations to see how the model behaves. Does it take much longer? How is the performance?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Want to learn more?\n",
    "\n",
    "Running deep learning programs usually needs a high performance platform. __PowerAI__ speeds up deep learning and AI. Built on IBM’s Power Systems, __PowerAI__ is a scalable software platform that accelerates deep learning and AI with blazing performance for individual users or enterprises. The __PowerAI__ platform supports popular machine learning libraries and dependencies including TensorFlow, Caffe, Torch, and Theano. You can use [PowerAI on IMB Cloud](https://cocl.us/ML0120EN_PAI).\n",
    "\n",
    "Also, you can use __Watson Studio__ to run these notebooks faster with bigger datasets.__Watson Studio__ is IBM’s leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, __Watson Studio__ enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of __Watson Studio__ users today with a free account at [Watson Studio](https://cocl.us/ML0120EN_DSX).This is the end of this lesson. Thank you for reading this notebook, and good luck on your studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Thanks for completing this lesson!\n",
    "\n",
    "This is the end of **Logistic Regression with TensorFlow** notebook. Hopefully, now you have a deeper understanding of Logistic Regression and how its structure and flow work. Thank you for reading this notebook and good luck on your studies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Created by: <a href=\"https://br.linkedin.com/in/walter-gomes-de-amorim-junior-624726121\">Saeed Aghabozorgi</a> , <a href=\"https://br.linkedin.com/in/walter-gomes-de-amorim-junior-624726121\">Walter Gomes de Amorim Junior</a> ,  Victor Barros Costa\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "Copyright &copy; 2018 [Cognitive Class](https://cocl.us/DX0108EN_CC). This notebook and its source code are released under the terms of the [MIT License](https://bigdatauniversity.com/mit-license/)."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [default]",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}