![](\"imgs/Perceptron.png\")
"
+ "![](\"../imgs/Perceptron.png\")
"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "356d5ec7-3392-4daa-9671-4cc7111c5c91"
},
@@ -216,8 +190,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -228,19 +200,14 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
- "![](\"imgs/MLP.png\")
"
+ "![](\"imgs/single_layer.png\")
\n",
+ "![](\"../imgs/single_layer.png\")
\n",
"\n",
"_(Source: Python Machine Learning, S. Raschka)_"
]
@@ -271,8 +235,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -304,10 +266,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"In **gradient descent optimization**, we update all the **weights simultaneously** after each epoch, and we define the _partial derivative_ for each weight $w_j$ in the weight vector $w$ as follows:\n",
"\n",
@@ -315,7 +274,7 @@
"\\frac{\\partial}{\\partial w_j} J(w) = \\sum_{i} ( y^{(i)} - a^{(i)} ) x^{(i)}_j\n",
"$$\n",
"\n",
- "**Note**: _The superscript $(i)$ refers to the ith sample. The subscript $j$ refers to the jth dimension/feature_\n",
+ "**Note**: _The superscript $(i)$ refers to the i-th sample. The subscript $j$ refers to the j-th dimension/feature_\n",
"\n",
"\n",
"Here $y^{(i)}$ is the target class label of a particular sample $x^{(i)}$ , and $a^{(i)}$ is the **activation** of the neuron \n",
@@ -359,8 +318,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -372,8 +329,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -397,7 +352,7 @@
"source": [
"### Perceptron and Adaline Models\n",
"\n",
- "Take a look at this notebook : ![](\"imgs/multi-layers-1.png\")
\n",
+ "![](\"../imgs/multi-layers-1.png\")
\n",
"\n",
"_(Source: Python Machine Learning, S. Raschka)_"
]
@@ -488,7 +443,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "![](\"imgs/multi-layers-2.png\")
\n",
+ "![](\"../imgs/multi-layers-2.png\")
\n",
"\n",
"_(Source: Python Machine Learning, S. Raschka)_"
]
@@ -522,7 +477,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "![](\"imgs/logistic_function.png\")
\n",
+ "![](\"../imgs/logistic_function.png\")
\n",
"\n",
"_(Source: Python Machine Learning, S. Raschka)_"
]
@@ -531,7 +486,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "![](\"imgs/fwd_step.png\")
\n",
+ "![](\"../imgs/fwd_step.png\")
\n",
"\n",
"_(Source: Python Machine Learning, S. Raschka)_"
]
@@ -540,7 +495,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "![](\"imgs/fwd_step_net.png\")
\n",
+ "![](\"../imgs/fwd_step_net.png\")
\n",
"\n",
"_(Source: Python Machine Learning, S. Raschka)_"
]
@@ -556,8 +511,6 @@
"cell_type": "markdown",
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"nbpresent": {
"id": "5678486b-caf4-440b-be62-2f1286982c71"
},
@@ -573,7 +526,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "![](\"imgs/bkwd_step_net.png\")
\n",
+ "![](\"../imgs/bkwd_step_net.png\")
\n",
"\n",
"_(Source: Python Machine Learning, S. Raschka)_"
]
@@ -581,8 +534,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -593,19 +544,14 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
- "![](\"imgs/backprop.png\")
"
+ "![](\"imgs/mlp_details.png\")
"
+ "![](\"../imgs/mlp_details.png\")
"
]
},
{
@@ -823,8 +745,6 @@
"execution_count": 8,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -842,8 +762,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -859,8 +777,6 @@
"execution_count": 9,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -875,8 +791,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -889,9 +803,7 @@
"cell_type": "code",
"execution_count": 10,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -904,8 +816,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -920,9 +830,7 @@
"cell_type": "code",
"execution_count": 11,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -934,8 +842,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -951,8 +857,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -988,8 +892,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1000,10 +902,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"```python\n",
"def activate(self, inputs):\n",
@@ -1037,8 +936,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1049,10 +946,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"```python\n",
"def backPropagate(self, targets, N, M):\n",
@@ -1103,9 +997,7 @@
"cell_type": "code",
"execution_count": 12,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1236,8 +1128,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1249,11 +1139,7 @@
{
"cell_type": "code",
"execution_count": 13,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -1276,8 +1162,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1289,11 +1173,7 @@
{
"cell_type": "code",
"execution_count": 14,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"name": "stdout",
@@ -1311,9 +1191,6 @@
"cell_type": "code",
"execution_count": 15,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1385,9 +1262,6 @@
"cell_type": "code",
"execution_count": 16,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -1411,8 +1285,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1426,8 +1298,6 @@
"execution_count": 17,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "skip"
}
@@ -1458,11 +1328,7 @@
{
"cell_type": "code",
"execution_count": 18,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"outputs": [
{
"data": {
@@ -1493,8 +1359,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1511,9 +1375,7 @@
"cell_type": "code",
"execution_count": 19,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -1525,23 +1387,19 @@
"cell_type": "code",
"execution_count": 20,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
- "# %load solutions/sol_111.py\n"
+ "# %load ../solutions/sol_111.py\n"
]
},
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1558,9 +1416,7 @@
"cell_type": "code",
"execution_count": 21,
"metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -1571,9 +1427,7 @@
"cell_type": "code",
"execution_count": 22,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"scrolled": false,
"slideshow": {
"slide_type": "subslide"
@@ -1581,14 +1435,12 @@
},
"outputs": [],
"source": [
- "# %load solutions/sol_112.py"
+ "# %load ../solutions/sol_112.py"
]
},
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -1596,7 +1448,7 @@
"source": [
"# Addendum\n",
"\n",
- "There is an additional notebook in the repo, i.e. [MLP and MNIST](extra/1.1.2 MLP and MNIST.ipynb) for a more complete (but still *naive* implementation) of **SGD** and **MLP** applied on **MNIST** dataset.\n",
+ "There is an additional notebook in the repo, i.e. [MLP and MNIST](1.1.2 MLP and MNIST.ipynb) for a more complete (but still *naive* implementation) of **SGD** and **MLP** applied on **MNIST** dataset.\n",
"\n",
"Another terrific reference to start is the online book http://neuralnetworksanddeeplearning.com/. Highly recommended! "
]
@@ -1619,7 +1471,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.5.2"
+ "version": "3.5.3"
},
"nbpresent": {
"slides": {
@@ -1944,5 +1796,5 @@
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
}
diff --git a/additional materials/1.1.1 Perceptron and Adaline.ipynb b/1. ANN/1.1.1 Perceptron and Adaline.ipynb
similarity index 98%
rename from additional materials/1.1.1 Perceptron and Adaline.ipynb
rename to 1. ANN/1.1.1 Perceptron and Adaline.ipynb
index 91b717e..64efe96 100644
--- a/additional materials/1.1.1 Perceptron and Adaline.ipynb
+++ b/1. ANN/1.1.1 Perceptron and Adaline.ipynb
@@ -2,20 +2,14 @@
"cells": [
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"##### (exceprt from Python Machine Learning Essentials, Supplementary Materials)"
]
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"### Sections\n",
"\n",
@@ -29,9 +23,7 @@
"cell_type": "code",
"execution_count": 1,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -43,20 +35,14 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"## Implementing a perceptron learning algorithm in Python"
]
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"[[back to top](#Sections)]"
]
@@ -65,9 +51,7 @@
"cell_type": "code",
"execution_count": 2,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -136,10 +120,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
+ "metadata": {},
"source": [
"![](\"imgs/tensorflow_head.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "# Tensorflow\n",
- "\n",
- ">**TensorFlow** (https://www.tensorflow.org/) is a software library, developed by Google Brain Team within Google's Machine Learning Intelligence research organization, for the purposes of conducting machine learning and deep neural network research. \n",
- "\n",
- ">TensorFlow combines the computational algebra of compilation optimization techniques, making easy the calculation of many mathematical expressions that would be difficult to calculate, instead.\n",
- "\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## Tensorflow Main Features"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "* Defining, optimizing, and efficiently calculating mathematical expressions involving multi-dimensional arrays (tensors).\n",
- "\n",
- "* Programming support of **deep neural networks** and machine learning techniques.\n",
- "\n",
- "* Transparent use of GPU computing, automating management and optimization of the same memory and the data used. You can write the same code and run it either on CPUs or GPUs. More specifically, TensorFlow will figure out which parts of the computation should be moved to the GPU.\n",
- "\n",
- "* High scalability of computation across machines and huge data sets.\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "source": [
- ">TensorFlow is available with Python and C++ support, but the **Python API** is better supported and much easier to learn."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "slide"
- }
- },
- "source": [
- "# Very Preliminary Example"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "11\n"
- ]
- }
- ],
- "source": [
- "# A simple calculation in Python\n",
- "x = 1\n",
- "y = x + 10\n",
- "print(y)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "outputs": [],
- "source": [
- "import tensorflow as tf"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Tensor(\"y/read:0\", shape=(), dtype=int32)\n"
- ]
- }
- ],
- "source": [
- "# The ~same simple calculation in Tensorflow\n",
- "x = tf.constant(1, name='x')\n",
- "y = tf.Variable(x+10, name='y')\n",
- "print(y)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "source": [
- "**Meaning**: \"When the variable `y` is computed, take the value of the constant `x` and add `10` to it\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "slide"
- }
- },
- "source": [
- "## Sessions and Models\n",
- "\n",
- "To actually calculate the value of the `y` variable and to evaluate expressions, we need to **initialise** the variables, and then create a **session** where the actual computation happens"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [],
- "source": [
- "model = tf.global_variables_initializer() # model is used by convention"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "11\n"
- ]
- }
- ],
- "source": [
- "with tf.Session() as session:\n",
- " session.run(model)\n",
- " print(session.run(y))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## Data Flow Graph"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "source": [
- "* (**IDEA**) \n",
- "_A Machine Learning application is the result of the repeated computation of complex mathematical expressions, thus \n",
- "we could describe this computation by using a **Data Flow Graph**"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "source": [
- "* **Data Flow Graph**: a graph where:\n",
- " - each Node represents the _instance_ of a mathematical operation \n",
- " - `multiply`, `add`, `divide`\n",
- " - each Edge is a multi-dimensional data set (`tensors`) on which the operations are performed."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## Tensorflow Graph Model"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "* **Node**: In TensorFlow, each node represents the instantion of an operation. \n",
- " - Each operation has inputs (`>= 2`) and outputs `>= 0`.\n",
- " \n",
- "* **Edges**: In TensorFlow, there are two types of edge:\n",
- " - Data Edges: \n",
- " They are carriers of data structures (`tensors`), where an output of one operation (from one node) becomes the input for another operation.\n",
- " - Dependency Edges: These edges indicate a _control dependency_ between two nodes (i.e. \"happens before\" relationship). \n",
- " + Let's suppose we have two nodes `A` and `B` and a dependency edge connecting `A` to `B`. This means that `B` will start its operation only when the operation in `A` ends. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## Tensorflow Graph Model (cont.)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "* **Operation**: This represents an abstract computation, such as adding or multiplying matrices. \n",
- " - An operation manages tensors, and It can just be polymorphic: the same operation can manipulate different tensor element types. \n",
- " + For example, the addition of two int32 tensors, the addition of two float tensors, and so on.\n",
- "\n",
- "* **Kernel**: This represents the concrete implementation of that operation. \n",
- " - A kernel defines the implementation of the operation on a particular device. \n",
- " + For example, an `add matrix` operation can have a CPU implementation and a GPU one."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## Tensorflow Graph Model Session"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "**Session**: When the client program has to establish communication with the TensorFlow runtime system, a session must be created. \n",
- " \n",
- "As soon as the session is created for a client, an initial graph is created and is empty. It has two fundamental methods:\n",
- "\n",
- "* `session.extend`: To be used during a computation, requesting to add more operations (nodes) and edges (data). The execution graph is then extended accordingly.\n",
- "\n",
- "* `session.run`: The execution graphs are executed to get the outputs (sometimes, subgraphs are executed thousands/millions of times using run invocations)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "slide"
- }
- },
- "source": [
- "# Tensorboard"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "**TensorBoard** is a visualization tool, devoted to analyzing Data Flow Graph and also to better understand the machine learning models. \n",
- "\n",
- "It can view different types of statistics about the parameters and details of any part of a computer graph graphically. It often happens that a graph of computation can be very complex."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## Tensorboard Example"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "Run the **TensorBoard** Server:"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "```shell\n",
- "tensorboard --logdir=/tmp/tf_logs\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "source": [
- "[Open TensorBoard](http://localhost:6006)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Example"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "95\n"
- ]
- }
- ],
- "source": [
- "a = tf.constant(5, name=\"a\")\n",
- "b = tf.constant(45, name=\"b\")\n",
- "y = tf.Variable(a+b*2, name='y')\n",
- "model = tf.global_variables_initializer()\n",
- "\n",
- "with tf.Session() as session:\n",
- " # Merge all the summaries collected in the default graph.\n",
- " merged = tf.summary.merge_all() \n",
- " \n",
- " # Then we create `SummaryWriter`. \n",
- " # It will write all the summaries (in this case the execution graph) \n",
- " # obtained from the code's execution into the specified path”\n",
- " writer = tf.summary.FileWriter(\"tmp/tf_logs_simple\", session.graph)\n",
- " session.run(model)\n",
- " print(session.run(y))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "slide"
- }
- },
- "source": [
- "# Data Types (Tensors)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## One Dimensional Tensor (Vector)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "tensor_1d = np.array([1, 2.5, 4.6, 5.75, 9.7])\n",
- "tf_tensor=tf.convert_to_tensor(tensor_1d,dtype=tf.float64)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[ 1. 2.5 4.6 5.75 9.7 ]\n",
- "1.0\n",
- "4.6\n"
- ]
- }
- ],
- "source": [
- "with tf.Session() as sess: \n",
- " print(sess.run(tf_tensor))\n",
- " print(sess.run(tf_tensor[0]))\n",
- " print(sess.run(tf_tensor[2]))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "## Two Dimensional Tensor (Matrix)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "[[ 0 1 2 3]\n",
- " [ 4 5 6 7]\n",
- " [ 8 9 10 11]\n",
- " [12 13 14 15]]\n",
- "[[ 0. 1. 2. 3.]\n",
- " [ 4. 5. 6. 7.]\n",
- " [ 8. 9. 10. 11.]\n",
- " [ 12. 13. 14. 15.]]\n"
- ]
- }
- ],
- "source": [
- "tensor_2d = np.arange(16).reshape(4, 4)\n",
- "print(tensor_2d)\n",
- "tf_tensor = tf.placeholder(tf.float32, shape=(4, 4))\n",
- "with tf.Session() as sess:\n",
- " print(sess.run(tf_tensor, feed_dict={tf_tensor: tensor_2d}))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "# Basic Operations (Examples)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "-"
- }
- },
- "outputs": [],
- "source": [
- "matrix1 = np.array([(2,2,2),(2,2,2),(2,2,2)],dtype='float32') \n",
- "matrix2 = np.array([(1,1,1),(1,1,1),(1,1,1)],dtype='float32')"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "tf_mat1 = tf.constant(matrix1) \n",
- "tf_mat2 = tf.constant(matrix2)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "outputs": [],
- "source": [
- "matrix_product = tf.matmul(tf_mat1, tf_mat2)\n",
- "matrix_sum = tf.add(tf_mat1, tf_mat2)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [],
- "source": [
- "matrix_det = tf.matrix_determinant(matrix2)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "outputs": [],
- "source": [
- "with tf.Session() as sess: \n",
- " prod_res = sess.run(matrix_product) \n",
- " sum_res = sess.run(matrix_sum) \n",
- " det_res = sess.run(matrix_det)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "matrix1*matrix2 : \n",
- " [[ 6. 6. 6.]\n",
- " [ 6. 6. 6.]\n",
- " [ 6. 6. 6.]]\n",
- "matrix1+matrix2 : \n",
- " [[ 3. 3. 3.]\n",
- " [ 3. 3. 3.]\n",
- " [ 3. 3. 3.]]\n",
- "det(matrix2) : \n",
- " 0.0\n"
- ]
- }
- ],
- "source": [
- "print(\"matrix1*matrix2 : \\n\", prod_res)\n",
- "print(\"matrix1+matrix2 : \\n\", sum_res)\n",
- "print(\"det(matrix2) : \\n\", det_res)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "source": [
- "# Handling Tensors"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "%matplotlib inline"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 17,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "import matplotlib.image as mp_image\n",
- "filename = \"imgs/keras-logo-small.jpg\"\n",
- "input_image = mp_image.imread(filename)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 18,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "fragment"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "input dim = 3\n",
- "input shape = (300, 300, 3)\n"
- ]
- }
- ],
- "source": [
- "#dimension\n",
- "print('input dim = {}'.format(input_image.ndim))\n",
- "#shape\n",
- "print('input shape = {}'.format(input_image.shape))"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 19,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "outputs": [
- {
- "data": {
- "image/png": "iVBORw0KGgoAAAANSUhEUgAAAQ4AAAEACAYAAABCu5jVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXmcXFW1779r73OqOwMJYcgcBCJX8SkXUUFwAlEEuV54\njgw+BZwHFJwAL2JwQlQUAUElooDz8BT1giDDE1FQvIigDAYQSAiZSUjSna5z9l7vj32604Ru0tVd\n3dXdWV/oT6pOnaqz6lSdX+291tpriapiGIbRCK7VBhiGMfYw4TAMo2FMOAzDaBgTDsMwGsaEwzCM\nhjHhMAyjYYZNOETkUBG5R0T+KSKnDNdxDMMYeWQ48jhExAH/BA4GlgK3Akep6j1NP5hhGCPOcI04\n9gUWqepDqloAPwSOGKZjGYYxwgyXcMwBFve6v6TaZhjGOGC4hEP62Ga57YYxTsiG6XWXALv0uj+X\n5OvoQURMSAyjRahqXz/uA2a4Rhy3Ak8XkaeJSA04CvjlljudfsYZFBqJZSSWgRgiUUNL/8745Bkt\nt8FsGj82tcKuMnZRD4GuGNEiEkOdIgZijEQtm3KBD8uIQ1WDiLwfuIYkTt9S1bu33E9QpATduI6u\nu++kkAzf4nFIsWQJHbf8qbVGbIHZNDBGo00w8naJlKjWKL2jsx7I2yLyzP+FnzAFXHPGCsM1VUFV\nfwM846l3AnHQeedf+eehR+DqBdJiV8hjoeT+y77XUhu2xGwaGKPRJhh5u5yDqI5SIy6vAXX+7fqr\nYd+XoU3yEAybcAyEAw96KTEKodYO9ToUdbTFwvE8QGNoqQ1bYjYNjNFoE4y8XelInjYCZVmQ5210\nMpkJCjIkz8ZmWppyftDLXoHPIr6MeGA0BF5e0GoD+sBsGhij0SYYebscEFHqgGik0Ei71vuOdQ7h\nGIZhGA1hwmEYRsOYcBiG0TAmHIZhNIwJh2EYDWPCYRhGw5hwGIbRMCYchmE0jAmHYRgNY8JhGEbD\nmHAYhtEwJhyGYTSMCYdhGA1jwmEYRsOYcBiG0TAmHIZhNIwJh2EYDWPCYRhGw5hwGIbRMCYchmE0\njAmHYRgNY8JhGEbDmHAYhtEwJhyGYTSMCYdhGA1jwmEYRsMMqXesiDwIrAMiUKjqviIyDfgR8DTg\nQeCNqrpuiHYahjGKGOqIIwIHqupzVXXfatupwLWq+gzgeuC0IR7DMIxRxlCFQ/p4jSOAS6vblwJH\nDvEYhmGMMoYqHApcLSK3isjbq20zVHU5gKouA3Ye4jEMwxhlDMnHARygqstEZGfgGhG5lyQmhmGM\nY4YkHNWIAlVdKSK/APYFlovIDFVdLiIzgRX9PX/BmQtAlGLxYubHyPOHYoxhGH1ya4z8aOFC8quu\nBmnO77qoDu6FRGQi4FR1g4hMAq4BzgQOBtao6tkicgowTVVP7eP5GoOCi3T88WbuP/BgtOgawlsx\nDAOS/6HEAREPaN7G02+8gYn77Q+iOHGoqgzlGEMZccwAfi4iWr3O91T1GhH5C/BjETkBeBh4w1AM\nNAxj9DFo4VDVfwF797F9DfCKoRi1rdA91pN+7vf3nEb2H+t0v9/e73sor9N9u5vxfO6Gk6E6R40h\noiJ4VcAREBwBBbSPr7QiqGi1v6BkKCVedFy6pB1QOhAcREUQ1IFqY+9XxBE14kTx6okIUSKicdhs\nH++YcLQQIfmq0tc3UooHPDUVoOxzf6fdc1hFKPEoXvvae+wTgVwzCi0rMVWcDsa/p/hKaSIRQcg0\nYrIxeEw4WkgEoiQxqAFeAyHJST9fau15XqhURxWijtMht0ChZXLwAbl4VIWSQGxgyKFEAuAERNIJ\nsy/+0LDz10IckKkjOKGLNHR2qgS0n8y8Sh6cw0cFYjUMT7/H440IOPGoBpwImzQg4hBtLHNREXLv\nKUMEiYgA6sDGHIPGVse2EAEyFxARatHhcBTiaSMjTVqe+CcoEYdGj0MQgY4Yifk41X+BUiPee4Km\nsyACDu3z/PT3BxBDpI0MUUE9bPJqsjEExuk3bvSx2aufYgQKxKyd/A1vYN7p72NCPhWngc5MaCsd\nQfrSdCE6IQ91Hr3gYjZcfDHTN21kXVE8Keow1CjESLJlxKM7ipLLJMpXvIR/+9LpbNhuGhOKGqpK\n9N17DgyvDpxS3H0vd334NLZf9HcmoFjW0OAx4Rgh1AsSHaoZjjpBhKmv+09mfXoBsvs8cs3wKmQO\nXCwR/+SPJkUTFI3K3E+cxlIKNlxwEU4hioI6MlFKIujYEQ8Bogio4gScZpRtNSa87vXsfu6nKXac\nw/YKwUW8Ck58Q29MY0zRqFmzmbHXs1m76O/46njG4LCpygiggA8O0YhQoHgmH/tG5p77Fdxuc8nV\nkWmaijhi8uL18zolELxDdtiJ2Wd+hmknfQiItKmiLhCJlCJk5H2GdEcrpSptro0ISK1k2mGvZuZZ\nZxB2mkMWHeoET8QN4j2JKkRPNrGNqe85Hg+UY+fUjEpMOEaAFFyVFEbNHJOPegPzz/syOmMWGTle\nHYUX6l4oY0ny3vX1OgLicF2RTCJuylRmnPM5Jp58MhsmTMLjUUl5C4ECN0YcpirQhiNoF5plcMCr\nmH7JuUycsys+CNEpGmOV5eIa9gOrS69BdEx+6Ytpe96+eDUfx1Aw4RghvEREFD9/PjPPPAO2n06O\n4FV6phU+KhnZU14XWQhILVT3FKcZT/vIh9npefsQo6IquBDTkH8E3lczEByOSNHextTXH80e//e7\ntE+dQxAFr4gEPEqkutgbHS1UUZjoBMlyZpx0EuJrY2g8NvoYK9+tMY0CwSlBoTZpCvmMuQSJ9EQF\nveJCgYuavP79faUFglckRlQ9USTlj+6wA7LD9qgTnCrgaTC5sqUEFQKe7Q89nJlf+jS17XZMPhtR\niBFRoXSekgykRz4GjIgiIaYEMhwTDj6I/H89x4RjCJhwjAAKSIQaGa7swmuJ1wwXIxILQHHeo5lQ\nZtDfT6pExUdPzNpBBOeUKCDq0SgECdUzHSBpWD+K6M42kV63FcgyT37Yocz7+leZMGse6iHJhIDP\ncHjyILQBXmPDF3xdAOfIVHEqZDtuz3avOqhfX5KxdUbXN2uc4kg/nvUqMTyKS/4H5xCfI+JJWRxC\nhvTkHjz5hQTx6XHnIMPhxaEOolPyAAEQ6pRIT5r1aEFFUMl6xMM5T2ybwKQ3HsUzL/kautNsJAqB\ngKahWBIJR4r/ieBcBn2GqvvHqSNKFV2JELOM9vm7IO0Tmv0WtxlMOIwRI1clKjipERyIC0w78CBm\nfW4BYfo8fHRERxU9aR6ZBpwEgs+IXthwzz0s+9olxE2bmniUbQvL4zBGjFS4pUwJ9a6NbL8DmHX5\nRbTtMBeJjlBFT5xzBDy+SYkodXHkXQIuUNz5N5a86Ri4775+VwQZW8dGHMaIkfJLoMw9O/7nkezx\no++T7ziP0kXUB5CAFyVq8t00ixwHLrDxlpt5+Li3ER/4Z4PL5IwtsRGHMWKoCl3ADgcezPSvnIWf\nOYMoZfJZBEVEKMWlJDBiyklpwm+bSJ2ND9zLsve8j66778JpVdvEpGPQmHAYTaf7ctxy/WnmM/ID\nX8a8S76JzJpBIUpW1cbA5anWSAA8xGolbEPH1d4xmyqVPQY6/nYvD73uP6k//CDQRl0DExDCE6w1\nGsGmKkbTSb/mm1PePY6YtzP5yNfy9G9fRJw1BzTHUUIUouQpWbYnesKgoidRI1oWBCJRQWPBut/9\njqXHH03H4odABRe7qAGlqI04hoAJh9F0cpRIxLmc4CBmkWkveSlzz/ksce5uSOkoneJk83rhZuDw\nBB/xEZxEOv9xL8vf/R7W33kXWZU92i0V0Ra4DQmbqhhNJ5IqqBYaIW8jP+ClzPzBxeTT5kAUNNtc\nNxVxTVvGK1FRXyOGks5b/sLDr30t5aplZCIQ0/jCIahgK2OHiI04jKZTItQArXmmveZI9rj8Utp2\nmkfIItHF5BAlphIBKcm1KYSqJMHjN/6Oh975TsKK1QQVYkwikXwuKWrj+llIaAwMG3EYTUdwdBKY\n9qKXMfPcz5PNnAlSpoI6muqGRC8IUhUe1n5XBDeC0zqb7rmf5e9+D/X77ycjI2rKoBURymqUkbn0\ne2nTlcFjwmEMmr6jJ4LLPBMPPoTdLvkmzJxOkFSAB4l0uZy8u2aogGoASSIy8OOmxYC972sIbPjz\n/7DoDUfhli4GgegK8uhSOwSqqvIihBhp4kBnm8SmKsagSVGTnNgTPRFCXmO7I1/Lbt84jzBzNmgO\nBFRT9KQd8JIGGEIqRuwa/RoqaCyJmqInJYH1v/s9S9/xDnTZEiBVjpeY2iGAIqrVmqH+CkEbjWDn\n0Bg0OUog4l2tip4o0/Z/EXPP+Ry6y+7DFj0RFQoXIUZKAuGu+3j07e9k4z33pBXETTyW0Tc2VTEG\nzROjJ+3UXnIgs77/DfIdhjd6gkCuOTEWdP7+Zpa+6U1sWrW8SgPZnABmHozhw0YcxqB5YvTkCJ5+\nycIUPfGR6EJP9ESbHD3R6ugbf3sdS97xTjauWY1XwWvl7yBNo2wJ2/BhIw5j0HRHT7bf78XMPO9s\nsumzQQq8ZqCxj+hJbDgbtE9iJ53/eIBH3vYOipWP0B5zSkhTIu3udrc57dxoPlv9FEXkWyKyXETu\n6LVtmohcIyL3isjVIjK112PnicgiEbldRJ7Uzd4Ye/Su3JXup/KGLsuYctjh7P6Dy2mfOZvgI6gH\nAnWfEX1yfIqkVlLa4FWsaBqtpK4QadVsUWft727mHwcdzKYVj6AKJQUiUjlKUzGj7iiKMTwMRP6/\nDbxqi22nAteq6jOA64HTAETkMGC+qu4BvAv4ehNtNVpE7+hJqmgKIcvZ7sgj2fXCrxBmzQbNEEK6\nwF2eyvwNNXqSFpz0ip4UPH7ttSx/93vwj61ANWWCpja6yTpP+lL3W0XNaApb/SRV9SbgsS02HwFc\nWt2+tLrfvf2y6nl/AqaKyIzmmGq0ihQ9CWS+jeAgZMq0fV/IvK+cjT5tPlKk6AmSpiRN+6UXR10i\nLgSUkvKOe1l2wjvpXLSoat4k1YpYY6QZ7IRzuqouB1DVZcD0avscYHGv/R6pthljmEgq51ePXUje\nTvvBhzPr55eTzZxLLQqSKXlVCBjnKn9Gc46bkRM1sP7a3/LgQQdSX/YIiuJFENWepK7x2XZ79NJs\n52hfPzb9fp4LzlwAohSLFzM/Rp7fZGOM5hAQcpR6zTH1P45gly99Cb/THIKUVTmc1ItB0yqy5o04\nFKLU6bjuOpaefDIb16+jhpBVvo9uulPHzafRN7fGyI8WLiS/6mqapeqDFY7lIjJDVZeLyExgRbV9\nCTCv135zgaX9vciCTy4AF+n4483cf/n30dDfnkYrEYRNwPbP25/ZX/si2U5zRiR64rSTjX+9m6X/\n53jqj60kjx7BEavGDwqVU9SE46l4gXMc/fa3M3G//UGUT535qSG/5kA/XeGJn8svgeOq28cBV/Ta\n/hYAEXkhsLZ7SmOMDXrX0Ird97Ocqf/xGnb/yQ9o23kmwTU7esIToieqEOubWHvltdz3soOpr15Z\nCVIAwhMcn1qlkFtC0siy1RGHiHwfOBDYUUQeBj4JfB74iYicADwMvAFAVa8UkVeLyH3ARuD44TLc\naD7JT+AJRJwkn0XpM7Z/zWuYc85nCbNm46MgUqLqiD5FT3r/pIgMIp5RtZ0L0p3xGVh/5dUsO/lk\nso51KbTay+lqiV2tZ6vCoarH9PPQK/rZ//1DsshoKRGYRM4m6pArU569D/O++iXi3HlI3RHy1Je2\nmdETIXVby0NJdELnbX/n0eOOI65bayIxSrERnrEZETyBLqlDljN5/1cy55c/xM2eRx4FySMZaSSi\nzuGadFVHBE9EQ2Tdr6/koZe/hLhu3eal8NV+vdtGGq3FhMPooQqKELxj+1e8ml0Wfp222U9LjZ5d\nQCXlZKpWBTWa+O0pUdZdfQ2PfvADdHV2bfav9NpHeLKzzWgNtlbF6EE0LVybuOfeTL/4PNzseXSx\nkfY4sSd6ok56Er2aFT2R0EnxP3fzyJuPJa7fQA2h9IqEJ44wTDBGDzbi2EbRXv92tzFwPmfyK17F\n7r/+GfmsuRCVdp2AujpFFT2RIUdPYk/kJCjErk7W/vc1LDrsUFi/HhWlJOJC/0lBNlVpPSYc2yAq\noOLQqnqFB0qXMfHQw9jlwi+j83bBq6S2jBGUdmpUa0/cUCp3VUvsiRQEohas/fVVLP3AB/FrV1EC\n6JO/lL2nJzZVGR3YVGUbxGnKhpgoykYBFWXSns9h3kXnoXPn4UshZBFX/bQ3OrLoD1VXrast8VHZ\n+PtbWXr0scSi60mdXG16MrqxEcc2SERwFHRIifiMKQcdxrwrf46bPZdaBLJIppp6kHiXusY3ARGo\nS0noKlj30yt44DWHEoo6sZosmUiMHUw4tlE8gjph6sGHMeeiC2ifuwvilOhSdiYSibHyYzTpio6S\nmiZtuPIaHv7IyRQdHT2RnC0xIRnd2FRlG0RQunBMfOZeTF94AdnsXYhsJIsTqjmCELxDPPgIEME1\nYe1J6KTjxv/hX296E1lQoqb66BF5Qu/43nkbJh6jExtxjHNUuteeSE/0BN/GlEMPY/df/4LarLlA\nIIsTUKkTvCd4XxXISZf1ILoX9ERPYlW5S+udrP3Jr7jntUfiyjqlFmnNiUhVHujJr2GMXkw4xjMC\niiOQISgZUIpn4qsOYd55X4Rd5qXoiaR2jFHayQQyASfVl8N119RqhFj9n/4jlqz+79+w5MMfprZ2\n9RNHEr2Wx1v0ZOxgU5VxTIqeOCYJdAiUTpn0zOewy0Xnp5BrFT3puUCb5ctQhyPiKNBSePyPf+bR\n178RjWVzDmC0HBtxjGNS9KSkQ0rwGVNefjjzrvoFbs68XtGTWOVsNDF6AqCBsKlk1Q9+zOLXHE6I\nwaYf4wgTjnGNVNETmHLQq5jztfNon7Nl9ER7oifNytdAInjhsZ/9kkdP+xjF+g1EaaQ7rDHasanK\nuCbSJY5Jz3wOMy75WhU96XhC9CR6B02Onki5iTVXXMeytx+P60pO0EIU1c1d1oyxjY04xgOVCCSn\no/REUsTXmHLoq9ntv39BbdY8ID4peiI90ZM+cr0HcFiN3ZW7IhqV2NXFqh/8jCVvO47Y1YUSCUQk\n0mf0xBib2IhjHKCiCEJBxgQCApQ4Jrzylcw99/PovHn4KEQXUwFgl6InT2AwI41YJ5DhAgSvqAgb\nf/VLHv3wKbBuTbKt2rV3noYx9jHhGAcIIFHYzkGXKiETJj1jL3b5+gXoLvPwpSNmsQp9NtHTIA5X\nCvUM2oo6635zJYvfcDRBgg1lxzn2+Y4LHE4iGymRLGPKKw5N0ZN5qXIXWcRrxFc+jaZ1OYse8VAr\nC9b+7McsefM78VU3N2N8Y8IxDtBUJQM8bPfSQ5hz3rlMmDMXQYmu6qZaRU+k2dETKQkIXQ+vZFOx\nsSoDaPGT8Y4JxzhANFIXYeIznsvMb19ENn8PSrrwUfDB4YMjkoHPcRFcbFKxUPFARDLP1LcezcRd\nn0XdK5axMf4x4RgPSKT98Fcz/9e/IJ87Fw9k2ob6vtaeNB496fewQIg1ANqmz2Dm979M26y5ZFRV\nxSSNPNTiKeMOE45xgOQ1ph9zLLW588hUqjUmgLT1rD2RnrUnzasyrIAXTZXANGP7vfdn+ic/issn\nUUoaeZRATTLEmXiMJ0w4xgMi5JMmoKJENLVEHIHinN21OlQipYDENqa89k1MOeLVuEyI4smpsUkL\nAmpftnGEfZbjBAVUBKn+RowoRBRfdWiqTdmJqR87kYmzdiOPASHi8dDEBk5G6zHhGA+IEDU+6coc\n7tUh0t2dXgU0IJVQTNpnP3b6zH+h4il8ABGcOuvKNo4w4RgHiCoZrup1klAd/p941ZSJWoYA4lEc\nqjlIjfY3H8sO73gH4Ai+RHr6shnjAROO8YKkdSM9d0diuuIAlDbJkwFeIAMnyqRYY+eTP8CkvZ7X\n02dWK2epiiP0VOrpVZnMGDNsVThE5FsislxE7ui17ZMiskREbqv+Du312GkiskhE7haRQ4bLcKMX\nksImzrseH8dICIfDIa6qTdoduakO2xUKanvswU4fPZG29gnUHSlJzQGSan9kCpnzFm0ZgwxkxPFt\n4FV9bP+yqu5T/f0GQET2BN4I7AkcBlwoI+qpM0YDApDnqV/L617HhDcexcSYlrnVIsQYUOcpgHos\nsaX2Y4+tCoeq3gQ81sdDfQnCEcAPVbVU1QeBRcC+Q7LQGHMoSlsZcZqT5xOY9/kF+P1fDCLUfVpZ\nWWgEJ2T05IkZY4ih+DjeJyK3i8hCEZlabZsDLO61zyPVNmMbIq2FcdX/Eb/zXHb+3Bm0zZpDTTMC\nSo1ePhkbcIw5BiscFwLzVXVvYBlwTrW9vz7BxjaE4Ci9EClB65QIU/d7EZOOPopSPG0IpSgiUIjF\nWsYig6rHoaore929GPhVdXsJMK/XY3OBpf29zoIzF4AoxeLFzI+R5w/GGKNhYtV00UUBCYASNQMX\ncSoEBK/dI4eQUsobnE9kQEARydLalfZ2Zpz8Xlbf9lfkhutwCrkKdbEiP8PNrTHyo4ULya+6mifE\n7IeA6ACKJ4jIrsCvVPU51f2Zqrqsun0y8AJVPUZEngV8D9iPNEX5LbCH9nEQEdEYFFyk4483c/+B\nB6NFV1Pe1GgkLX1Xanvtxfwbf082ZTKCa8nPbSQSk4uBQiEnsvGm38GkqUx+7nMpSsjyKrGLVGHM\nydCreGiAjffexr0vPpTssZVVTmnobvtkNAlHqgAHEQ9o3sbTb7yBifvtD6I4cagOrbjCQMKx3wf+\nCPybiDwsIscDXxCRO0TkduBlwMkAqnoX8GPgLuBK4L19iYbRWiSA1wBlAUWdDT/6CfceeDhrPvMF\nutasJM8E0dCdx9684zpo23MfZn7mDNyE7dAsojiiTVbGHFudqqjqMX1s/vZT7H8WcNZQjDKGGedQ\nhQJY991vs+Ljn8BLF2tuuIa2X1/JDm8+Di8Z4kCq7ixNoSqivvPr38Cm/3cjHT/9KeqEQpXMfl7G\nFJY5ug0igKhj7S9+zKqTT4EVK9AItccfZ81JH6fjoX+CE4QA2kT/gwQKFLfDNKafchJ+4o5ojE1r\nBGWMHCYc2yBl10ZWfP87LH/j8WzqWEuZ1q9SqqLrlrPiPR9G16xB1dEVmrfGREVpQ4m+xuR9XsiU\nL38GJk7BAm9jDxOOcUwESlLXeCUV1tGii9XfuYxHP3QqQoEPoKQVrApsFCH8/jpWfvtiYlFQ8z7V\n74qaFs5FQEFDiTbo0hRxaUGeRlQdM044lqlHvR4lI1JFYlxaxyLVJMkYnZhwjGNEI76EEEpQxQVY\nfvl3WPrRTyCrlvdc9mmtWlpq1oZDNnWx6usLWf/AIiQ6ogbKqkhQt28EcY2PE2IV1q30QP0kZn30\nvUx+7r9TtmfJjhjJtarxYSORUYsJx3jHB7wXwsYuHvn2N1j9tvfiN6zuc9fk+wgUZJQPPMDKT3yK\netdGUu95JRKryuapEEfDNb0kjXyipFFQFgTdYy+mfeBE8pjTRVoIF7MczyCEyRgxTDjGNQ5Vh5aB\n1QsvZM1pZxDov6BOBIIIdRdQETb97OesPfdcNEby6PB4IoE8Jqepa/TK1pRV6hScU5wouWZs//rX\ns90JbyGKQwXKWKcQrNjgKMY+mXGMAqV41lx4Pqv/63TC6pVEtGpr8GRcVQwoRKVdHeICa8+7gPU3\n/ZF6yifCiQMCuLJxD4QAKgRAYolSgA8wcSKzv3wOE194ADVVsijkNtwY1ZhwjGc6HmflV7/Mwx8+\njXpHFx5P6UOq3NUHkQgZTALqKETP+lUrWfnFc/ArFhMdqDqiCKhvODesFEWjImREl4HLAAGJ1Nom\nsvMZp1DMmkkbjtKRpkXGqMSEYzygCjH5DUKEGKAIdR47/zxWf/osspBGB0ED7UGe0umoQUmrVwIl\nkEel44YbWHPZjwElRkE01dpodFAgVcVi13NP8OoQTZmq2734Zez0lhMoJOIEgkKoOs95IIojOou0\njAZMOMYFkUJSPwRHgQgs+tKXePhTZ6FrVlVLyJLHIDzFgjIBXNVZwSm4bgnpXM+jn/4iHbf+CXHp\nYhap5i4N4HGIF1xVKcx1N3sRQZyQTZrM7I9+kPIZz4IQaXNZyiitbCo1Imqej9GAfQbjgCipX2uh\ngWJjJ0u+dBblxxeQd3Y07RjZ+hWsOukUyuUP4pTUuyU2xxEhlER1gBB2mM78n1yCmzGbUku8q+HJ\nKUV6mmWHphzVGAomHOMAV110vnSsPP9rPHb2F8hioGhWaS2BAmHDbX9m5cXfQTu7AGnKilkAF9Lr\niCq5RrZ7xvPZ4dSPIJOm0BXrFFLiVak61VqYdhRgwjEOUJJPY9nZZ7FmwafgsXWs981bA+LwqaZG\nvWDNhZfQ8cAdSCGUzXl5gk9JZiqBKIHOTJh+9LHUXnIAHmhTxUmq3C6VgBitxYRjHKCxZNVXzmfV\ngtPRok6BMKGJP8uqgULSCtawYjGPnnQa5cZlTbyAFXGp21uIgo/gd9qReaefSjFrD6ITugSiZnhv\nsjEaMOEYDxQFm37/e3xMIVWnkRCb18dNABTqoniBzutv5KGzvoLWN6BEygBoTGtZgtK40zQlk4l4\nMvHUXHKWtr/wAOZ/7gy6XI6PgvpIDFYtbDRgwjEeEI+K9jgNU/Oj5vkCPGk6FFQRhYySjZf+mLU3\n3ABdqQFTQUQJFA4aLi4lDueqfjBOqjYxDi852x91ODuf8FYEyENoWuk7Y2iYcBhbJS2s9+RkoFA4\nyFY8xMpzz6dY/SiFpOZKMUq1YrY5Yx0JULRNZdr738eEZ+/NpsxVXeEsl6PVmHAYWyU5QZUMIYij\nLlB3Sv2GG1n2ja9RIyDRI87hS6o+sUNHfSSLMOnf9mT7D30gNdbO0njKaC0mHMZWSV+SCERKFSYE\nj48eLeus/+w5rL/uWuq+agbruyunD526BATF5Rk7v/UYdnzrcUi52SKjddgnYGwVR5UtSiAj4CsP\nSogCWuepabQQAAAVT0lEQVShj54O/7pv8+J7zZtyXE+WKhyLomTs9oVzkBe+IK2pMVqKCYexVVL1\njTRlEVJVsUDEoXT4jHDXXaxaeDGxszP5H2KAkAr+hJgCLRq14e4pXtOoQ6smCvUdpjL71NORWXMp\nyStPR5Yetz6SI4oJhzEgvG72LHQLiYiSFyVZfRNLz/sm6/5xG5AWopUuhYUFpRykT0Ji6rwiUcki\nFJTscMjL2P51/xtx6VUjJRme3LpwjCgmHMagKdQnhylKrWMtK9/3MerLl6acDDQJh5Ygoaou1miY\nNrVqipJW0k4gp94+kdn/9VHanvVsohfEAdSJTUp/NwaGCYcxaByRIGnRWd3Bpr/8mUdPO5O4qQMJ\nPtXscI4cRQfjMK1W0QanBA8Zjjw6dMZsdv/OQmoz5yIKnQ6izVRGFBMOY9BkKJkqtTzHBYd3kQ1X\n/IzHr7mG0kciQsCniuXSeFO4qCDRUSsURyqW7JyQq6O29/PZ+YPvJ6eGy9qQGC2/YwQx4TAGjZIc\noGUZ0iK0AJs2rmX1F8/DLfkn6mJVuVyQ6BteDi/OEQWiT6UGA4GQih8i3jH12DdTe/Wh5PUuoq8K\ndxgjggmHMWgiVK2NI2hKec+LyPpb/sCjZ1+AU8VpanuvhMaLG1O5OUQQl5Hhu+v+EDWSzZjFTgtO\ngUlT6W51W1VEBVKfFgvcDg8mHMag6c7h9JUztCfaoiWrv/YdVv38F6CR6FKBZNdgDVEhtUtwUq1j\nEUlNndBqO0zd5wXMvuRc2qUGQEfmaCMD53rCxzaBaT4D6VY/V0SuF5G7ROROEflAtX2aiFwjIveK\nyNUiMrXXc84TkUUicruI7D2cb8AYfaSoykZWnf5JOhbdk3wTUZs2k4gEpFoJrOKZcsRRTHnLscS8\nRl4qBQEiFOJSn5bmHNboxUBGHCXwIVV9FrA/8D4ReSZwKnCtqj4DuB44DUBEDgPmq+oewLuArw+L\n5caoJQB4CA8sYsVF38B1laiDpi1+w4NLBZUlQp61sd0H30O22+7glRytKqRLNdcxms1WhUNVl6nq\n7dXtDcDdwFzgCODSardLq/tU/15W7f8nYKqIzGiy3cYoxuMgdlcMW8jKP92EaKOdZvtH1UGMqWCy\nB3UFk/bamxmnnkgWJ6CSwsOZE6JaB9rhoCEfh4jsCuwN3ALMUNXlkMQFmF7tNgdY3Otpj1TbjG0E\nQVEVglNqZSfr3nEinQ8tal5hIVclrysETa0oM8nY+dg3k7/trdQ1pz1CEUpzcAwTAxYOEZkM/BT4\nYDXyeKoq+1ticbJtitTCIIqgAp3338PS0z+HPr4KNFBHCTGiQVENoI0FagXAOZx4PCBkaVu2HbM/\nfToTD9iPqDCx8m/Yl6/5ZAPZSUQykmhcrqpXVJuXi8gMVV0uIjOBFdX2JcC8Xk+fCyzt63UXnLkA\nRCkWL2Z+jDx/UG/BGG0UwEQyQlSilkQHG678NauufhU7v/ZocCkdve49WemImdDYetoUUdl8DxBB\nFSbPmMGcj32IB0/4J2HNCmqaU1Bs0wOPW2PkRwsXkl91ddMqqIkOYHGQiFwGrFLVD/XadjawRlXP\nFpFTge1V9VQReTXwPlU9XEReCJyrqi/s4zU1BgUX6fjjzdx/4MFo0dWUNzUaUQSHUttrL+bf+Huy\nKZMRXFOG0rGrzgNvfD0bf/mrnl/XiCOj8RWpzSAKoJ6AIyeAj7gg+H32YZef/4TJc3ellNTsKY0Z\nXFN8mEFTGny5voNHTv8Ea87/KuIcEsM2JRxpJXPKsPGA5m08/cYbmLjf/iCKE4c2XN/xycd4SkTk\nRcCxwMtF5K8icpuIHAqcDbxSRO4FDgY+D6CqVwL/EpH7gG8A7x2KgcbYwykEAt6l9asuwCandN1+\nG0tOPYPY1YWPDleGKkOrOW5TASQ4/OTJzDjjw0zcf3+yGCxZaRjY6lRFVf8A/VbCf0U/z3n/UIwy\nxj41wMeSoppCTIkZm1xB1w++y9pXHsLktxxDntVSnQ7p/wvWEAKdXmmjoH3KHOYt/AoPvPhw9LFV\nzXh1oxcmxkbT0Srns5DU+qAUpaREo6AOFn/+c4S/3ZkWsaXW0005blRoj0qIHjJom78XO5x6Ejpx\nYlNe39iMCYcxLGjVKTpqmroUKEGUKDXk/vtYfdEFlEWdKA7RAEFRTdmGGgti2XiBwAzACZlLviNX\na2PaW08g3/8FeByO9IXvynPLJh0iJhxG00kdXbTHIdm9XiRXwYU6xJKll3yXTVddlUogqxJ9SUnA\na0TEoyINO3a795fK0ypA+84zmP/FsyimTKEEHBlZEciqtS3G4DDhMEaMiMOJ4IB27eLhd32Qzn/8\nDUGQIKSF96mOh3M0vJq2WzB6RwpFhOzf92XmOWfD5KnUKcldRtSieW9sG8SEwxgxogScKl0KAaFY\nu5gVpy8gPrYqrZ4NnohP9TYG0Yuuv9QCr46d3nAM0444EvEOYkEUayU5FEw4jBEjq7qw5d4j6qjV\nYf0NN7DmJ1fQpQWF91W+C0RR4iCSO6SP59R9STZhEjt9+EQm7b57KgdgqjEkTDiMEUMROkVTzwQi\nIuDWd7D6ggth0R2gqRShUEdUGi9uTN+jjixmSKa07fXvbP/RUyiYgLNVs0PChMMYMRSt2iyk/0oF\niHTddQcPvfsTsOnxpCnkCHFQlTR6jzi6b3uELgmIOHZ885uZ9u7jEfUIWbc7lUiGer9NZZgOBRMO\nY8RI2R2JzV+8VF6w83fXsfKb38ZpnQIh4Br2QfREU6pqYZu3KbWQpW507TVmf+Jk8v2ehydQ+pRF\nUq8FXAgwiONui5hwGKMAh7iCx8+/kI1/uQWqRk5Nq1qukSgpHBw0wox57HzSSegOO5CpUDhPrdQq\n08MYCHamjJZTElF1dD50P2vO/SYSA6qOIa7D6iE6T5QSDQUeT6Y50/7zCCb9nzfhIzgCuXoKFB1M\nReVtEBMOo+Vk4hGNEJXHf/JTll2ykEID0qQl4FUbOcT7qrdLRNsnMOeM/6L27OejkVSNPc8oLNwy\nIEw4jJajGkA8qhC1zpqPfJyuW25qmq9BiWQxJ2WHRKIXMiK1abOY8fWz8PN2pS4RKQtqakXDBoIJ\nx4ihFEDuhBAdqFSthRp7jRBLtFQ0NYRHA8QYyaLS6UDJUte0LFYZEaMfBUqN5EDdQ75+Hcs++wXK\nFUvQIlAqRA0QFQ0BbbC1k0MQlxymThy+KmIsClNesD87vu0E6m2TyIEg3taxDICx8c0aB+QitONY\nD6jrIkrjQ/EYlQJHPQMk4KqGzsFB4RztQBuQK7hAw2s9WoakJrEByCOo82y64SYe++H/RTPBqSJl\nZJNLq2sb7SUpSFXwXFJ6exV1KVyB+HZ2OOa1TJ2zM50K7Wr1OwaCnaMRQIEuVbp8ZLsI7dGnVaAN\nXtcOR00F0ZCuHRFUFR+VPHraI3RRUjpHWpw6Nn47nSqTKlNVoYOAdHWw4ivns/EffwMCZebJguLK\nplW/w6sQUNbd9nfqKx/HC3SJ1SgdCCYcI0RWDY2DCkEVp1nD4b8kFmlILzgInhgzxEGZRQrAu9RS\nJBtDn2zpYD1VGwNxTIypZWTXww/xr6NPQDaswanDR6VwoE3K+nQxg8fX0/H971E8vhG7HAaOnakR\nQBCCOjKFjcuX8tiNNxBifVDjAZHk4EsqouDrrL/1D3Teew8RqGsamdSb/B6GExfBZY4aCqrUUXIF\nT0T+fjuPnvM1KDdRz0qc8zStU4oT1tx8A+uuvgqRLlRT1Q5zjm4dE44RIKIIkUIdbsVKln3kNDqu\nuIJGB8VKKsiLOjRGgivZeNP/45H3n0q8dxEZHl+5ROuud0WM0Y3gcLFy9go4yelE8BopHDz2ne/S\ncdPNKA4XtWm5nfUN61l95meResADGQWqY8Yz1FJMOEaAqkoEOZHSK+5f93Hfu07k8Z99i1JjCkOG\nSMkmQlknpjo4T/4jIhJRPOojxXXX89Drjqe88zaIkZK0bD1SMqGEsTJbVyISIwEQVVRLPEpwGT5C\nXPIgK8/6Am1hA6VUPo4G/so0kCEGJaqimkoZPn7ZRay9425yTetmRMaQQ7nFDKivijF01FVtCxXK\nCFNXr+H+kz7Nbm4y7a86nLZYAwmIRoqs7HPALMTqYnBs+MvNPHLiR3Crl1GoPOFXONWzGDvIk26n\nsZJS4oFCIo9ddz1tXzyfnT5wInVpa6idglOlFMGFlKSxyQeyB1fyyDd+SL6pg4BQutRESiSN5oyn\nxoRjBBDAxfRb1gZ0ShplTHj0EZZ/9HTyX12J8xNQF8hDSBe+SnXxbL6wogiinugyOm/5A8U9d4FX\nYpSGq2WNBbIIiEOip6aBR878AsU995K5NqJsnoh1n6Mt/938mFI6IVNHJFJ4gSUryO5bhEMIEpkA\nJL0YhydyGDDhGAEUKKWaF0aY6GCTRDaKkP1rMcUD3wMfCZoullT76slfYBGhdFAL1etJRj0UbIey\naaTf1AiwyXnqMbCdOELmybo28Pjl3yPSWEhWSJ1bvDiixlSwOAMCRAcTSOe9DqioaccAMOEYIXLS\nPDtU3+IcqKGUFOQ4QhAKSY7C0E9OqaqSB4gInoxCSyY4YYMq2Tj8sudR2Q7o0pJQwgQ8AQEtG45I\nSe4pikibeLo0kJdJyIODGJJolDWPFKFJzRrGNyYcI4SQft00pT5S+O45t1KIEqOSq1BISqd2/WRH\nllV9CVc5EGMcvx9ilEhdoRBPJpGoIbU2FJcWpQ2QTJSuym8RNeWBeCCoIupJBZIVV1jW6EAZr9+5\nUYWQvqRUvU0VkmhAyvDUWN1URDc7CPt8rS0umPHsxhOtphga0mgNBlUVrFTw1fmOBHz1WilQlYS6\nWkBrDBATjhFiy/FDf/e3FiwYS9GSodLXOWnG+x/ouTb6x0ZmhmE0zEC61c8VketF5C4RuVNETqy2\nf1JEllTd67s72Hc/5zQRWSQid4vIIcP5BgzDGHkGMlUpgQ+p6u0iMhn4HxH5bfXYl1X1y713FpE9\ngTcCewJzgWtFZA/tr1uOYRhjjq2OOFR1mareXt3eANwNzKke7muaeATwQ1UtVfVBYBGwb3PMNQxj\nNNCQj0NEdgX2Bv5UbXqfiNwuIgtFZGq1bQ6wuNfTHmGz0BiGMQ4YsHBU05SfAh+sRh4XAvNVdW9g\nGXBO9659PN2mKYYxjhhQOFZEMpJoXK6qVwCo6speu1wM/Kq6vQSY1+uxucDSvl53wZkLUgLU4sXM\nj5HnN2i8YRhb59YY+dHCheRXXd20ZBUZiM9SRC4DVqnqh3ptm6mqy6rbJwMvUNVjRORZwPeA/UhT\nlN8CT3KOiojGoOAiHX+8mfsPPBgtuprypgxjW8ZByrAl4gHN23j6jTcwcb/9QRQnDh1i05qtjjhE\n5EXAscCdIvJX0rTj48AxIrI3KbnvQeBdAKp6l4j8GLgLKID3WkTFMMYXWxUOVf0D9Lnu5zdP8Zyz\ngLOGYJdhGKMYyxw1DKNhTDgMw2gYEw7DMBrGhMMwjIYx4TAMo2FMOAzDaBgTDsMwGsaEwzCMhjHh\nMAyjYUw4DMNoGBMOwzAaxoTDMIyGMeEwDKNhTDgMw2gYEw7DMBrGhMMwjIYx4TAMo2FMOAzDaBgT\nDsMwGsaEwzCMhjHhMAyjYUw4DMNoGBMOwzAaxoTDMIyGGVDv2OEjggai1hEXCK03yDDGPKWAUwEy\nlBInDonNHSO09DoNKjj1eD+JUjMyoCS20iTDGPN4gQIh14B6iETEhaYeo6XC4VQJTimJiEDhSsR0\nwzCGRIiONiKgSISQOaIMqcf0k2ipcJReyFVoKwQXHBI9QnOV0TC2ORzUY6Qd6ATEO1zR3L7vLRWO\nPETUBWLWRayBczVUbchhGENCHJlEigjOZ4gCbWVzD6HaXCUa8IFF9Nprr+PlBx5EZ+c64j/+hoZJ\nOOotsaebG2+7jZfus09LbdgSs2lgjEabYOTtUnE4AhHFB4fmBfme/0623fYgihOHqg5p7tLSEceN\nN/6Ol7/8ICZOmgr7vpQAOJo7F2uUW66+hkP3P6ClNmyJ2TQwRqNNMIrsauIgocXRTyUSK7EQQGmx\nboBo+htNmE0DYzTaBKPKrtikqGVrhaMooKMTtIoxizZTFAdHvUA3dLTYiC0wmwbGaLQJRo1dgqBN\nEo6W+jhacmDDMIbs42iZcBiGMXaxtSqGYTSMCYdhGA3TEuEQkUNF5B4R+aeInNIKGyo7HhSRv4nI\nX0Xkz9W2aSJyjYjcKyJXi8jUEbDjWyKyXETu6LWtXztE5DwRWSQit4vI3iNo0ydFZImI3Fb9Hdrr\nsdMqm+4WkUOGyaa5InK9iNwlIneKyAeq7S07V33YdGK1vaXnathR1RH9I4nVfcDTgBy4HXjmSNtR\n2fIAMG2LbWcDH6tunwJ8fgTseDGwN3DH1uwADgP+u7q9H3DLCNr0SeBDfey7J/BXUpRu1+rzlWGw\naSawd3V7MnAv8MxWnqunsKml52q4/1ox4tgXWKSqD6lqAfwQOKIFdkDKGtnyHBwBXFrdvhQ4criN\nUNWbgMe2YscRvbZfVj3vT8BUEZkxQjZB35k2RwA/VNVSVR8EFpE+52bbtExVb69ubwDuBubSwnPV\nj01zqodbdq6Gm1YIxxxgca/7S9h8okcaBa4WkVtF5O3VthmquhzSlwLYuUW2Td/CjunV9i3P3yOM\n7Pl7XzXsX9hrSjDiNonIrqQR0S08+TNrybnqZdOfqk2j4lwNB60Qjr5UuFUx4QNU9fnAq0kf8kta\naMtAaeX5uxCYr6p7A8uAc1phk4hMBn4KfLD6le/vWCNmVx82jYpzNVy0QjiWALv0uj8XWNoCO7p/\nnVDVlcAvSEPG5d3DWRGZCaxohW1PYccSYF6v/Ubs/KnqSq0m6sDFbB5ij5hNIpKRLtDLVfWKanNL\nz1VfNo2GczWctEI4bgWeLiJPE5EacBTwy5E2QkQmVr8SiMgk4BDgzsqW46rd3gpc0ecLDINJPPHX\nqLcdx/Wy45fAWwBE5IXA2u5h+nDbVF2U3bwW+Hsvm44SkZqI7AY8HfjzMNl0CXCXqn6117ZWn6sn\n2TRKztXw0QqPLHAoyfu8CDi1RTbsRoro/JUkGKdW23cArq3s+y2w/QjY8n3Sr04X8DBwPDCtPzuA\nC0je+L8B+4ygTZcBd1Tn7Rck30L3/qdVNt0NHDJMNr0ICL0+t9uq71K/n9lwn6unsKml52q4/yzl\n3DCMhrHMUcMwGsaEwzCMhjHhMAyjYUw4DMNoGBMOwzAaxoTDMIyGMeEwDKNhTDgMw2iY/w819DdL\nuqg9fwAAAABJRU5ErkJggg==\n",
- "text/plain": [
- "![](\"imgs/keras-logo-small.jpg\"){kind=logo}
\n",
- "\n",
- "## Keras: Deep Learning library for Theano and TensorFlow"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.\n",
- "ref: https://keras.io/"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Why this name, Keras?\n",
- "\n",
- "Keras (κέρας) means _horn_ in Greek. It is a reference to a literary image from ancient Greek and Latin literature, first found in the _Odyssey_, where dream spirits (_Oneiroi_, singular _Oneiros_) are divided between those who deceive men with false visions, who arrive to Earth through a gate of ivory, and those who announce a future that will come to pass, who arrive through a gate of horn. It's a play on the words κέρας (horn) / κραίνω (fulfill), and ἐλέφας (ivory) / ἐλεφαίρομαι (deceive).\n",
- "\n",
- "Keras was initially developed as part of the research effort of project ONEIROS (Open-ended Neuro-Electronic Intelligent Robot Operating System).\n",
- "\n",
- ">_\"Oneiroi are beyond our unravelling --who can be sure what tale they tell? Not all that men look for comes to pass. Two gates there are that give passage to fleeting Oneiroi; one is made of horn, one of ivory. The Oneiroi that pass through sawn ivory are deceitful, bearing a message that will not be fulfilled; those that come out through polished horn have truth behind them, to be accomplished for men who see them.\"_ Homer, Odyssey 19. 562 ff (Shewring translation)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "### Kaggle Challenge Data (again)\n",
- "\n",
- "See: [Data Description](1.2 Introduction - Tensorflow.ipynb#kaggle)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using TensorFlow backend.\n"
- ]
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "9 classes\n",
- "93 dims\n"
- ]
- }
- ],
- "source": [
- "from kaggle_data import load_data, preprocess_data, preprocess_labels\n",
- "\n",
- "X_train, labels = load_data('data/kaggle_ottogroup/train.csv', train=True)\n",
- "X_train, scaler = preprocess_data(X_train)\n",
- "Y_train, encoder = preprocess_labels(labels)\n",
- "\n",
- "X_test, ids = load_data('data/kaggle_ottogroup/test.csv', train=False)\n",
- "\n",
- "X_test, _ = preprocess_data(X_test, scaler)\n",
- "\n",
- "nb_classes = Y_train.shape[1]\n",
- "print(nb_classes, 'classes')\n",
- "\n",
- "dims = X_train.shape[1]\n",
- "print(dims, 'dims')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "---"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "## Hands On - Keras Logistic Regression\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "from keras.models import Sequential\n",
- "from keras.layers import Dense, Activation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "93 dims\n",
- "Building model...\n",
- "9 classes\n",
- "Epoch 1/10\n",
- "61878/61878 [==============================] - 2s - loss: 1.0577 \n",
- "Epoch 2/10\n",
- "61878/61878 [==============================] - 3s - loss: 0.7702 \n",
- "Epoch 3/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.7283 \n",
- "Epoch 4/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.7075 \n",
- "Epoch 5/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6945 \n",
- "Epoch 6/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6854 \n",
- "Epoch 7/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6787 \n",
- "Epoch 8/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6735 \n",
- "Epoch 9/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6694 \n",
- "Epoch 10/10\n",
- "61878/61878 [==============================] - 2s - loss: 0.6659 \n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "The core data structure of Keras is a model, a way to organize layers. The main type of model is the Sequential model, a linear stack of layers.\n" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "What we did here is stacking a Fully Connected (Dense) layer of trainable weights from the input to the output and an Activation layer on top of the weights layer." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "##### Dense" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": true, - "deletable": true, - "editable": true - }, - "source": [ - "```python\n", - "from keras.layers.core import Dense\n", - "\n", - "Dense(units, activation=None, use_bias=True, \n", - " kernel_initializer='glorot_uniform', bias_initializer='zeros', \n", - " kernel_regularizer=None, bias_regularizer=None, \n", - " activity_regularizer=None, kernel_constraint=None, bias_constraint=None)\n", - "```" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "* `units`: int > 0.\n", - "\n", - "* `init`: name of initialization function for the weights of the layer (see initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.\n", - "\n", - "* `activation`: name of activation function to use (see activations), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. \"linear\" activation: a(x) = x).\n", - "\n", - "* `weights`: list of Numpy arrays to set as initial weights. The list should have 2 elements, of shape (input_dim, output_dim) and (output_dim,) for weights and biases respectively.\n", - "\n", - "* `kernel_regularizer`: instance of WeightRegularizer (eg. L1 or L2 regularization), applied to the main weights matrix.\n", - "\n", - "* `bias_regularizer`: instance of WeightRegularizer, applied to the bias.\n", - "\n", - "* `activity_regularizer`: instance of ActivityRegularizer, applied to the network output.\n", - "\n", - "* `kernel_constraint`: instance of the constraints module (eg. maxnorm, nonneg), applied to the main weights matrix.\n", - "\n", - "* `bias_constraint`: instance of the constraints module, applied to the bias.\n", - "\n", - "* `use_bias`: whether to include a bias (i.e. make the layer affine rather than linear)." - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "## (some) others `keras.core.layers`" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "* `keras.layers.core.Flatten()`\n", - "* `keras.layers.core.Reshape(target_shape)`\n", - "* `keras.layers.core.Permute(dims)`\n", - "\n", - "```python\n", - "model = Sequential()\n", - "model.add(Permute((2, 1), input_shape=(10, 64)))\n", - "# now: model.output_shape == (None, 64, 10)\n", - "# note: `None` is the batch dimension\n", - "```\n", - "\n", - "* `keras.layers.core.Lambda(function, output_shape=None, arguments=None)`\n", - "* `keras.layers.core.ActivityRegularization(l1=0.0, l2=0.0)`" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "
![](\"imgs/dl_overview.png\")
\n",
- "\n",
- "Credits: Yam Peleg ([@Yampeleg](https://twitter.com/yampeleg))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "##### Activation"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "source": [
- "```python\n",
- "from keras.layers.core import Activation\n",
- "\n",
- "Activation(activation)\n",
- "```"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "**Supported Activations** : [https://keras.io/activations/]\n",
- "\n",
- "**Advanced Activations**: [https://keras.io/layers/advanced-activations/]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "##### Optimizer"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).\n",
- "Here we used SGD (stochastic gradient descent) as an optimization algorithm for our trainable weights. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "![](\"http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif\")
\n",
- "\n",
- "Source & Reference: http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "\"Data Sciencing\" this example a little bit more\n",
- "====="
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "What we did here is nice, however in the real world it is not useable because of overfitting.\n",
- "Lets try and solve it with cross validation."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "##### Overfitting"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "In overfitting, a statistical model describes random error or noise instead of the underlying relationship. Overfitting occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. \n",
- "\n",
- "A model that has been overfit has poor predictive performance, as it overreacts to minor fluctuations in the training data."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "source": [
- "\n",
- "![](\"imgs/overfitting.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "To avoid overfitting, we will first split out data to training set and test set and test out model on the test set.\n", - "Next: we will use two of keras's callbacks EarlyStopping and ModelCheckpoint" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "---" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Let's see first the model we implemented" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": { - "collapsed": false - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "_________________________________________________________________\n", - "Layer (type) Output Shape Param # \n", - "=================================================================\n", - "dense_2 (Dense) (None, 9) 846 \n", - "_________________________________________________________________\n", - "activation_2 (Activation) (None, 9) 0 \n", - "=================================================================\n", - "Total params: 846.0\n", - "Trainable params: 846.0\n", - "Non-trainable params: 0.0\n", - "_________________________________________________________________\n" - ] - } - ], - "source": [ - "model.summary()" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": { - "collapsed": true - }, - "outputs": [], - "source": [ - "from sklearn.model_selection import train_test_split\n", - "from keras.callbacks import EarlyStopping, ModelCheckpoint" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": { - "collapsed": false, - "deletable": true, - "editable": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Train on 52596 samples, validate on 9282 samples\n", - "Epoch 1/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6634 - val_loss: 0.6561\n", - "Epoch 2/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6626 - val_loss: 0.6562\n", - "Epoch 3/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6619 - val_loss: 0.6562\n", - "Epoch 4/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6613 - val_loss: 0.6562\n", - "Epoch 5/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6607 - val_loss: 0.6561\n", - "Epoch 6/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6601 - val_loss: 0.6557\n", - "Epoch 7/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6596 - val_loss: 0.6554\n", - "Epoch 8/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6591 - val_loss: 0.6551\n", - "Epoch 9/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6586 - val_loss: 0.6550\n", - "Epoch 10/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6582 - val_loss: 0.6548\n", - "Epoch 11/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6577 - val_loss: 0.6545\n", - "Epoch 12/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6572 - val_loss: 0.6544\n", - "Epoch 13/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6568 - val_loss: 0.6541\n", - "Epoch 14/20\n", - "52596/52596 [==============================] - 1s - loss: 0.6563 - val_loss: 0.6538\n", - "Epoch 15/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6559 - val_loss: 0.6534\n", - "Epoch 16/20\n", - "52596/52596 [==============================] - 1s - loss: 0.6555 - val_loss: 0.6533\n", - "Epoch 17/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6551 - val_loss: 0.6534\n", - "Epoch 18/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6548 - val_loss: 0.6529\n", - "Epoch 19/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6544 - val_loss: 0.6525\n", - "Epoch 20/20\n", - "52596/52596 [==============================] - 0s - loss: 0.6540 - val_loss: 0.6523\n" - ] - }, - { - "data": { - "text/plain": [ - "
\n", - "$$b_t = b_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial b}$$\n", - "\n", - "where $\\eta$ is the learning rate.\n", - "\n", - "**NOTE:** Recall that **linear regression** is indeed a **simple neuron** with a linear activation function!!" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Placeholders and variables\n", - "To implement and run this simple model, we will use the [Keras backend module](http://keras.io/backend/), which provides an abstraction over Theano and Tensorflow, two popular tensor manipulation libraries that provide automatic differentiation.\n", - "\n", - "First of all, we define the necessary variables and placeholders for our computational graph. Variables maintain state across executions of the computational graph, while placeholders are ways to feed the graph with external data.\n", - "\n", - "For the linear regression example, we need three variables: `w`, `b`, and the learning rate for SGD, `lr`. Two placeholders `x` and `target` are created to store $x_i$ and $y_i$ values." - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": { - "collapsed": false, - "deletable": true, - "editable": true - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Using TensorFlow backend.\n" - ] - } - ], - "source": [ - "import keras.backend as K\n", - "import numpy as np\n", - "\n", - "# Placeholders and variables\n", - "x = K.placeholder()\n", - "target = K.placeholder()\n", - "lr = K.variable(0.1)\n", - "w = K.variable(np.random.rand())\n", - "b = K.variable(np.random.rand())" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "collapsed": true, - "deletable": true, - "editable": true - }, - "source": [ - "## Model definition\n", - "Now we can define the $y = w\\cdot x + b$ relation as well as the MSE loss in the computational graph." - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": { - "collapsed": true, - "deletable": true, - "editable": true - }, - "outputs": [], - "source": [ - "# Define model and loss\n", - "y = w * x + b\n", - "loss = K.mean(K.square(y-target))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "Then, given the gradient of MSE wrt to `w` and `b`, we can define how we update the parameters via SGD:" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": { - "collapsed": true, - "deletable": true, - "editable": true - }, - "outputs": [], - "source": [ - "grads = K.gradients(loss, [w,b])\n", - "updates = [(w, w-lr*grads[0]), (b, b-lr*grads[1])]" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "The whole model can be encapsulated in a `function`, which takes as input `x` and `target`, returns the current loss value and updates its parameter according to `updates`." - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": { - "collapsed": false, - "deletable": true, - "editable": true - }, - "outputs": [], - "source": [ - "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "## Training\n", - "Training is now just a matter of calling the `function` we have just defined. Each time `train` is called, indeed, `w` and `b` will be updated using the SGD rule.\n", - "\n", - "Having generated some random training data, we will feed the `train` function for several epochs and observe the values of `w`, `b`, and loss." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": { - "collapsed": false, - "deletable": true, - "editable": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Loss: 0.045, w, b: [0.48, 0.38]\n", - "Loss: 0.010, w, b: [0.64, 0.41]\n", - "Loss: 0.006, w, b: [0.72, 0.37]\n", - "Loss: 0.003, w, b: [0.78, 0.34]\n", - "Loss: 0.002, w, b: [0.82, 0.31]\n", - "Loss: 0.001, w, b: [0.86, 0.30]\n", - "Loss: 0.001, w, b: [0.88, 0.28]\n", - "Loss: 0.000, w, b: [0.90, 0.27]\n", - "Loss: 0.000, w, b: [0.92, 0.26]\n", - "Loss: 0.000, w, b: [0.93, 0.26]\n" - ] - } - ], - "source": [ - "# Generate data\n", - "np_x = np.random.rand(1000)\n", - "np_target = 0.96*np_x + 0.24\n", - "\n", - "# Training\n", - "loss_history = []\n", - "for epoch in range(200):\n", - " current_loss = train([np_x, np_target])[0]\n", - " loss_history.append(current_loss)\n", - " if epoch % 20 == 0:\n", - " print(\"Loss: %.03f, w, b: [%.02f, %.02f]\" % (current_loss, K.eval(w), K.eval(b)))" - ] - }, - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "We can also plot the loss history:" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": { - "collapsed": false, - "deletable": true, - "editable": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "[
![](\"../imgs/tensorflow_head.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "subslide"
+ }
+ },
+ "source": [
+ "# Tensorflow\n",
+ "\n",
+ ">**TensorFlow** (https://www.tensorflow.org/) is a software library, developed by Google Brain Team within Google's Machine Learning Intelligence research organization, for the purposes of conducting machine learning and deep neural network research. \n",
+ "\n",
+ ">TensorFlow combines the computational algebra of compilation optimization techniques, making easy the calculation of many mathematical expressions that would be difficult to calculate, instead.\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "subslide"
+ }
+ },
+ "source": [
+ "## Tensorflow Main Features"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "* Defining, optimizing, and efficiently calculating mathematical expressions involving multi-dimensional arrays (tensors).\n",
+ "\n",
+ "* Programming support of **deep neural networks** and machine learning techniques.\n",
+ "\n",
+ "* Transparent use of GPU computing, automating management and optimization of the same memory and the data used. You can write the same code and run it either on CPUs or GPUs. More specifically, TensorFlow will figure out which parts of the computation should be moved to the GPU.\n",
+ "\n",
+ "* High scalability of computation across machines and huge data sets.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "fragment"
+ }
+ },
+ "source": [
+ ">TensorFlow is available with Python and C++ support, but the **Python API** is better supported and much easier to learn."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "slideshow": {
+ "slide_type": "slide"
+ }
+ },
+ "source": [
+ "# Very Preliminary Example"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {
+ "slideshow": {
+ "slide_type": "subslide"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "11\n"
+ ]
+ }
+ ],
+ "source": [
+ "# A simple calculation in Python\n",
+ "x = 1\n",
+ "y = x + 10\n",
+ "print(y)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": true,
+ "slideshow": {
+ "slide_type": "subslide"
+ }
+ },
+ "outputs": [],
+ "source": [
+ "import tensorflow as tf"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "slideshow": {
+ "slide_type": "fragment"
+ }
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "![](\"../imgs/keras-logo-small.jpg\"){kind=logo}
\n",
+ "\n",
+ "## Keras: Deep Learning library for Theano and TensorFlow"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ ">Keras is a minimalist, highly modular neural networks library, written in Python and capable of running on top of either TensorFlow or Theano. \n",
+ "\n",
+ ">It was developed with a focus on enabling fast experimentation. Being able to go from idea to result with the least possible delay is key to doing good research.\n",
+ "ref: https://keras.io/"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "### Kaggle Challenge Data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ ">The Otto Group is one of the world’s biggest e-commerce companies, A consistent analysis of the performance of products is crucial. However, due to diverse global infrastructure, many identical products get classified differently.\n",
+ "For this competition, we have provided a dataset with 93 features for more than 200,000 products. The objective is to build a predictive model which is able to distinguish between our main product categories. \n",
+ "Each row corresponds to a single product. There are a total of 93 numerical features, which represent counts of different events. All features have been obfuscated and will not be defined any further.\n",
+ "\n",
+ "https://www.kaggle.com/c/otto-group-product-classification-challenge/data"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### For this section we will use the Kaggle Otto Group Challenge Data. You will find these data in \n",
+ "`../data/kaggle_ottogroup/` folder."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Logistic Regression\n",
+ "\n",
+ "This algorithm has nothing to do with the canonical _linear regression_, but it is an algorithm that allows us to solve problems of classification (supervised learning). \n",
+ "\n",
+ "In fact, to estimate the dependent variable, now we make use of the so-called **logistic function** or **sigmoid**. \n",
+ "\n",
+ "It is precisely because of this feature we call this algorithm logistic regression.\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Data Preparation"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Using TensorFlow backend.\n"
+ ]
+ }
+ ],
+ "source": [
+ "from kaggle_data import load_data, preprocess_data, preprocess_labels\n",
+ "import numpy as np\n",
+ "import matplotlib.pyplot as plt"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "9 classes\n",
+ "93 dims\n"
+ ]
+ }
+ ],
+ "source": [
+ "X_train, labels = load_data('../data/kaggle_ottogroup/train.csv', train=True)\n",
+ "X_train, scaler = preprocess_data(X_train)\n",
+ "Y_train, encoder = preprocess_labels(labels)\n",
+ "\n",
+ "X_test, ids = load_data('../data/kaggle_ottogroup/test.csv', train=False)\n",
+ "X_test, _ = preprocess_data(X_test, scaler)\n",
+ "\n",
+ "nb_classes = Y_train.shape[1]\n",
+ "print(nb_classes, 'classes')\n",
+ "\n",
+ "dims = X_train.shape[1]\n",
+ "print(dims, 'dims')"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array(['Class_1', 'Class_2', 'Class_3', 'Class_4', 'Class_5', 'Class_6',\n",
+ " 'Class_7', 'Class_8', 'Class_9'], dtype=object)"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "np.unique(labels)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "array([[ 0., 0., 0., ..., 0., 0., 0.],\n",
+ " [ 0., 0., 0., ..., 1., 0., 0.],\n",
+ " [ 0., 0., 0., ..., 0., 0., 0.],\n",
+ " ..., \n",
+ " [ 0., 0., 0., ..., 0., 0., 0.],\n",
+ " [ 0., 0., 0., ..., 0., 0., 0.],\n",
+ " [ 0., 0., 1., ..., 0., 0., 0.]])"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "Y_train # one-hot encoding"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Using Theano"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "import theano as th\n",
+ "import theano.tensor as T"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Epoch 1\n",
+ "Epoch 2\n",
+ "Epoch 3\n",
+ "Epoch 4\n",
+ "Epoch 5\n",
+ "Epoch 6\n",
+ "Epoch 7\n",
+ "Epoch 8\n",
+ "Epoch 9\n",
+ "Epoch 10\n",
+ "target values for Data:\n",
+ "[ 0. 0. 0. ..., 0. 0. 0.]\n",
+ "prediction on training set:\n",
+ "[ True True False ..., True True True]\n"
+ ]
+ }
+ ],
+ "source": [
+ "#Based on example from DeepLearning.net\n",
+ "rng = np.random\n",
+ "N = 400\n",
+ "feats = 93\n",
+ "training_steps = 10\n",
+ "\n",
+ "# Declare Theano symbolic variables\n",
+ "x = T.matrix(\"x\")\n",
+ "y = T.vector(\"y\")\n",
+ "w = th.shared(rng.randn(feats), name=\"w\")\n",
+ "b = th.shared(0., name=\"b\")\n",
+ "\n",
+ "# Construct Theano expression graph\n",
+ "p_1 = 1 / (1 + T.exp(-T.dot(x, w) - b)) # Probability that target = 1\n",
+ "prediction = p_1 > 0.5 # The prediction thresholded\n",
+ "xent = -y * T.log(p_1) - (1-y) * T.log(1-p_1) # Cross-entropy loss function\n",
+ "cost = xent.mean() + 0.01 * (w ** 2).sum() # The cost to minimize\n",
+ "gw, gb = T.grad(cost, [w, b]) # Compute the gradient of the cost\n",
+ " \n",
+ "\n",
+ "# Compile\n",
+ "train = th.function(\n",
+ " inputs=[x,y],\n",
+ " outputs=[prediction, xent],\n",
+ " updates=((w, w - 0.1 * gw), (b, b - 0.1 * gb)),\n",
+ " allow_input_downcast=True)\n",
+ "predict = th.function(inputs=[x], outputs=prediction, allow_input_downcast=True)\n",
+ "\n",
+ "#Transform for class1\n",
+ "y_class1 = []\n",
+ "for i in Y_train:\n",
+ " y_class1.append(i[0])\n",
+ "y_class1 = np.array(y_class1)\n",
+ "\n",
+ "# Train\n",
+ "for i in range(training_steps):\n",
+ " print('Epoch %s' % (i+1,))\n",
+ " pred, err = train(X_train, y_class1)\n",
+ "\n",
+ "print(\"target values for Data:\")\n",
+ "print(y_class1)\n",
+ "print(\"prediction on training set:\")\n",
+ "print(predict(X_train))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Using Tensorflow"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "import tensorflow as tf"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# Parameters\n",
+ "learning_rate = 0.01\n",
+ "training_epochs = 25\n",
+ "display_step = 1"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "# tf Graph Input\n",
+ "x = tf.placeholder(\"float\", [None, dims]) \n",
+ "y = tf.placeholder(\"float\", [None, nb_classes])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "The core data structure of Keras is a model, a way to organize layers. The main type of model is the Sequential model, a linear stack of layers.\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "What we did here is stacking a Fully Connected (Dense) layer of trainable weights from the input to the output and an Activation layer on top of the weights layer." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "##### Dense" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "```python\n", + "from keras.layers.core import Dense\n", + "\n", + "Dense(units, activation=None, use_bias=True, \n", + " kernel_initializer='glorot_uniform', bias_initializer='zeros', \n", + " kernel_regularizer=None, bias_regularizer=None, \n", + " activity_regularizer=None, kernel_constraint=None, bias_constraint=None)\n", + "```" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* `units`: int > 0.\n", + "\n", + "* `init`: name of initialization function for the weights of the layer (see initializations), or alternatively, Theano function to use for weights initialization. This parameter is only relevant if you don't pass a weights argument.\n", + "\n", + "* `activation`: name of activation function to use (see activations), or alternatively, elementwise Theano function. If you don't specify anything, no activation is applied (ie. \"linear\" activation: a(x) = x).\n", + "\n", + "* `weights`: list of Numpy arrays to set as initial weights. The list should have 2 elements, of shape (input_dim, output_dim) and (output_dim,) for weights and biases respectively.\n", + "\n", + "* `kernel_regularizer`: instance of WeightRegularizer (eg. L1 or L2 regularization), applied to the main weights matrix.\n", + "\n", + "* `bias_regularizer`: instance of WeightRegularizer, applied to the bias.\n", + "\n", + "* `activity_regularizer`: instance of ActivityRegularizer, applied to the network output.\n", + "\n", + "* `kernel_constraint`: instance of the constraints module (eg. maxnorm, nonneg), applied to the main weights matrix.\n", + "\n", + "* `bias_constraint`: instance of the constraints module, applied to the bias.\n", + "\n", + "* `use_bias`: whether to include a bias (i.e. make the layer affine rather than linear)." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## (some) others `keras.core.layers`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "* `keras.layers.core.Flatten()`\n", + "* `keras.layers.core.Reshape(target_shape)`\n", + "* `keras.layers.core.Permute(dims)`\n", + "\n", + "```python\n", + "model = Sequential()\n", + "model.add(Permute((2, 1), input_shape=(10, 64)))\n", + "# now: model.output_shape == (None, 64, 10)\n", + "# note: `None` is the batch dimension\n", + "```\n", + "\n", + "* `keras.layers.core.Lambda(function, output_shape=None, arguments=None)`\n", + "* `keras.layers.core.ActivityRegularization(l1=0.0, l2=0.0)`" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "
![](\"../imgs/dl_overview.png\")
\n",
+ "\n",
+ "Credits: Yam Peleg ([@Yampeleg](https://twitter.com/yampeleg))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Activation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "collapsed": true
+ },
+ "source": [
+ "```python\n",
+ "from keras.layers.core import Activation\n",
+ "\n",
+ "Activation(activation)\n",
+ "```"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Supported Activations** : [https://keras.io/activations/]\n",
+ "\n",
+ "**Advanced Activations**: [https://keras.io/layers/advanced-activations/]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Optimizer"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "If you need to, you can further configure your optimizer. A core principle of Keras is to make things reasonably simple, while allowing the user to be fully in control when they need to (the ultimate control being the easy extensibility of the source code).\n",
+ "Here we used SGD (stochastic gradient descent) as an optimization algorithm for our trainable weights. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "Source & Reference: http://sebastianruder.com/content/images/2016/09/saddle_point_evaluation_optimizers.gif"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\"Data Sciencing\" this example a little bit more\n",
+ "====="
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "What we did here is nice, however in the real world it is not useable because of overfitting.\n",
+ "Lets try and solve it with cross validation."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "##### Overfitting"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "In overfitting, a statistical model describes random error or noise instead of the underlying relationship. Overfitting occurs when a model is excessively complex, such as having too many parameters relative to the number of observations. \n",
+ "\n",
+ "A model that has been overfit has poor predictive performance, as it overreacts to minor fluctuations in the training data."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "![](\"../imgs/overfitting.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "To avoid overfitting, we will first split out data to training set and test set and test out model on the test set.\n", + "Next: we will use two of keras's callbacks EarlyStopping and ModelCheckpoint" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "---" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Let's see first the model we implemented" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "_________________________________________________________________\n", + "Layer (type) Output Shape Param # \n", + "=================================================================\n", + "dense_1 (Dense) (None, 9) 846 \n", + "_________________________________________________________________\n", + "activation_1 (Activation) (None, 9) 0 \n", + "=================================================================\n", + "Total params: 846\n", + "Trainable params: 846\n", + "Non-trainable params: 0\n", + "_________________________________________________________________\n" + ] + } + ], + "source": [ + "model.summary()" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "from keras.callbacks import EarlyStopping, ModelCheckpoint" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Train on 52596 samples, validate on 9282 samples\n", + "Epoch 1/50\n", + "52596/52596 [==============================] - 1s - loss: 1.6516 - val_loss: 1.6513\n", + "Epoch 2/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6501 - val_loss: 1.6499\n", + "Epoch 3/50\n", + "52596/52596 [==============================] - 1s - loss: 1.6488 - val_loss: 1.6486\n", + "Epoch 4/50\n", + "52596/52596 [==============================] - 1s - loss: 1.6474 - val_loss: 1.6473\n", + "Epoch 5/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6462 - val_loss: 1.6461\n", + "Epoch 6/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6449 - val_loss: 1.6448\n", + "Epoch 7/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6437 - val_loss: 1.6437\n", + "Epoch 8/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6425 - val_loss: 1.6425\n", + "Epoch 9/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6414 - val_loss: 1.6414\n", + "Epoch 10/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6403 - val_loss: 1.6403\n", + "Epoch 11/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6392 - val_loss: 1.6393\n", + "Epoch 12/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6382 - val_loss: 1.6383\n", + "Epoch 13/50\n", + "52596/52596 [==============================] - 1s - loss: 1.6372 - val_loss: 1.6373\n", + "Epoch 14/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6362 - val_loss: 1.6363\n", + "Epoch 15/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6352 - val_loss: 1.6354\n", + "Epoch 16/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6343 - val_loss: 1.6345\n", + "Epoch 17/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6334 - val_loss: 1.6336\n", + "Epoch 18/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6325 - val_loss: 1.6327\n", + "Epoch 19/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6316 - val_loss: 1.6319\n", + "Epoch 20/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6308 - val_loss: 1.6311\n", + "Epoch 21/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6300 - val_loss: 1.6303\n", + "Epoch 22/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6292 - val_loss: 1.6295\n", + "Epoch 23/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6284 - val_loss: 1.6287\n", + "Epoch 24/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6276 - val_loss: 1.6280\n", + "Epoch 25/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6269 - val_loss: 1.6273\n", + "Epoch 26/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6262 - val_loss: 1.6265\n", + "Epoch 27/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6254 - val_loss: 1.6258\n", + "Epoch 28/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6247 - val_loss: 1.6252\n", + "Epoch 29/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6241 - val_loss: 1.6245\n", + "Epoch 30/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6234 - val_loss: 1.6238\n", + "Epoch 31/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6227 - val_loss: 1.6232\n", + "Epoch 32/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6221 - val_loss: 1.6226\n", + "Epoch 33/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6215 - val_loss: 1.6220\n", + "Epoch 34/50\n", + "52596/52596 [==============================] - 1s - loss: 1.6209 - val_loss: 1.6214\n", + "Epoch 35/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6203 - val_loss: 1.6208\n", + "Epoch 36/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6197 - val_loss: 1.6202\n", + "Epoch 37/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6191 - val_loss: 1.6197\n", + "Epoch 38/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6186 - val_loss: 1.6191\n", + "Epoch 39/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6180 - val_loss: 1.6186\n", + "Epoch 40/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6175 - val_loss: 1.6181\n", + "Epoch 41/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6170 - val_loss: 1.6175\n", + "Epoch 42/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6165 - val_loss: 1.6170\n", + "Epoch 43/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6160 - val_loss: 1.6166\n", + "Epoch 44/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6155 - val_loss: 1.6161\n", + "Epoch 45/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6150 - val_loss: 1.6156\n", + "Epoch 46/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6145 - val_loss: 1.6151\n", + "Epoch 47/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6141 - val_loss: 1.6147\n", + "Epoch 48/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6136 - val_loss: 1.6142\n", + "Epoch 49/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6132 - val_loss: 1.6138\n", + "Epoch 50/50\n", + "52596/52596 [==============================] - 0s - loss: 1.6127 - val_loss: 1.6134\n" + ] + }, + { + "data": { + "text/plain": [ + "
![](\"../imgs/MLP.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Forward and Backward Propagation"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![](\"../imgs/backprop.png\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Q:** _How hard can it be to build a Multi-Layer Fully-Connected Network with keras?_\n",
+ "\n",
+ "**A:** _It is basically the same, just add more layers!_"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "_________________________________________________________________\n",
+ "Layer (type) Output Shape Param # \n",
+ "=================================================================\n",
+ "dense_2 (Dense) (None, 100) 9400 \n",
+ "_________________________________________________________________\n",
+ "dense_3 (Dense) (None, 9) 909 \n",
+ "_________________________________________________________________\n",
+ "activation_2 (Activation) (None, 9) 0 \n",
+ "=================================================================\n",
+ "Total params: 10,309\n",
+ "Trainable params: 10,309\n",
+ "Non-trainable params: 0\n",
+ "_________________________________________________________________\n"
+ ]
+ }
+ ],
+ "source": [
+ "model = Sequential()\n",
+ "model.add(Dense(100, input_shape=(dims,)))\n",
+ "model.add(Dense(nb_classes))\n",
+ "model.add(Activation('softmax'))\n",
+ "model.compile(optimizer='sgd', loss='categorical_crossentropy')\n",
+ "model.summary()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 24,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Train on 52596 samples, validate on 9282 samples\n",
+ "Epoch 1/20\n",
+ "52596/52596 [==============================] - 1s - loss: 1.2113 - val_loss: 0.8824\n",
+ "Epoch 2/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.8229 - val_loss: 0.7851\n",
+ "Epoch 3/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.7623 - val_loss: 0.7470\n",
+ "Epoch 4/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.7329 - val_loss: 0.7258\n",
+ "Epoch 5/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.7143 - val_loss: 0.7107\n",
+ "Epoch 6/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.7014 - val_loss: 0.7005\n",
+ "Epoch 7/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6918 - val_loss: 0.6922\n",
+ "Epoch 8/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6843 - val_loss: 0.6868\n",
+ "Epoch 9/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6784 - val_loss: 0.6817\n",
+ "Epoch 10/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6736 - val_loss: 0.6773\n",
+ "Epoch 11/20\n",
+ "52596/52596 [==============================] - 0s - loss: 0.6695 - val_loss: 0.6739\n",
+ "Epoch 12/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6660 - val_loss: 0.6711\n",
+ "Epoch 13/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6631 - val_loss: 0.6688\n",
+ "Epoch 14/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6604 - val_loss: 0.6670\n",
+ "Epoch 15/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6582 - val_loss: 0.6649\n",
+ "Epoch 16/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6563 - val_loss: 0.6626\n",
+ "Epoch 17/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6545 - val_loss: 0.6611\n",
+ "Epoch 18/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6528 - val_loss: 0.6598\n",
+ "Epoch 19/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6514 - val_loss: 0.6578\n",
+ "Epoch 20/20\n",
+ "52596/52596 [==============================] - 1s - loss: 0.6500 - val_loss: 0.6571\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "\n", + "$$b_t = b_{t-1} - \\eta \\frac{\\partial \\mathcal{l}}{\\partial b}$$\n", + "\n", + "where $\\eta$ is the learning rate.\n", + "\n", + "**NOTE:** Recall that **linear regression** is indeed a **simple neuron** with a linear activation function!!" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Definition: Placeholders and Variables" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First of all, we define the necessary variables and placeholders for our computational graph. Variables maintain state across executions of the computational graph, while placeholders are ways to feed the graph with external data.\n", + "\n", + "For the linear regression example, we need three variables: `w`, `b`, and the learning rate for SGD, `lr`. \n", + "\n", + "Two placeholders `x` and `target` are created to store $x_i$ and $y_i$ values." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Placeholders and variables\n", + "x = K.placeholder()\n", + "target = K.placeholder()\n", + "w = K.variable(np.random.rand())\n", + "b = K.variable(np.random.rand())" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "#### Notes:\n", + "\n", + "In case you're wondering what's the difference between a **placeholder** and a **variable**, in short:\n", + "\n", + "* Use `K.variable()` for trainable variables such as weights (`W`) and biases (`b`) for your model.\n", + "* Use `K.placeholder()` to feed actual data (e.g. training examples)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "collapsed": true + }, + "source": [ + "## Model definition\n", + "Now we can define the $y = w\\cdot x + b$ relation as well as the MSE loss in the computational graph." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Define model and loss" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# %load ../solutions/sol_2311.py" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Then, given the gradient of MSE wrt to `w` and `b`, we can define how we update the parameters via SGD:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "# %load ../solutions/sol_2312.py" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "The whole model can be encapsulated in a `function`, which takes as input `x` and `target`, returns the current loss value and updates its parameter according to `updates`." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "train = K.function(inputs=[x, target], outputs=[loss], updates=updates)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## Training\n", + "Training is now just a matter of calling the `function` we have just defined. Each time `train` is called, indeed, `w` and `b` will be updated using the SGD rule.\n", + "\n", + "Having generated some random training data, we will feed the `train` function for several epochs and observe the values of `w`, `b`, and loss." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Generate data\n", + "np_x = np.random.rand(1000)\n", + "np_target = 0.96*np_x + 0.24" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Training\n", + "loss_history = []\n", + "for epoch in range(200):\n", + " current_loss = train([np_x, np_target])[0]\n", + " loss_history.append(current_loss)\n", + " if epoch % 20 == 0:\n", + " print(\"Loss: %.03f, w, b: [%.02f, %.02f]\" % (current_loss, K.eval(w), K.eval(b)))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "We can also plot the loss history:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# Plot loss history" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "collapsed": true + }, + "outputs": [], + "source": [ + "# %load ../solutions/sol_2313.py" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### Final Note:\n", + "\n", + "Please switch back your backend to `tensorflow` before moving on. It may be useful for next notebooks !-)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.5.3" + } + }, + "nbformat": 4, + "nbformat_minor": 1 +} diff --git a/kaggle_data.py b/2. Deep Learning Frameworks/kaggle_data.py similarity index 100% rename from kaggle_data.py rename to 2. Deep Learning Frameworks/kaggle_data.py diff --git a/mnist_data.py b/2. Deep Learning Frameworks/mnist_data.py similarity index 100% rename from mnist_data.py rename to 2. Deep Learning Frameworks/mnist_data.py diff --git a/2.2.1 CNN HandsOn - MNIST & FC Nets.ipynb b/2.2.1 CNN HandsOn - MNIST & FC Nets.ipynb deleted file mode 100644 index feb5bc0..0000000 --- a/2.2.1 CNN HandsOn - MNIST & FC Nets.ipynb +++ /dev/null @@ -1,254 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": { - "deletable": true, - "editable": true - }, - "source": [ - "# Fully Connected Feed-Forward Network\n", - "\n", - "In this notebook we will play with Feed-Forward FC-NN (Fully Connected Neural Network) for a *classification task*: Image Classification on MNIST Dataset\n", - "\n", - "**RECALL**\n", - "\n", - "In the FC-NN, the output of each layer is computed using the activations from the previous one, as follows:\n", - "\n", - "$$h_{i} = \\sigma(W_i h_{i-1} + b_i)$$\n", - "\n", - "where ${h}_i$ is the activation vector from the $i$-th layer (or the input data for $i=0$), ${W}_i$ and ${b}_i$ are the weight matrix and the bias vector for the $i$-th layer, respectively. \n", - "
- \n",
- "
- axis: Integer, the axis that should be normalized\n",
- " (typically the features axis).\n",
- " For instance, after a
Conv2Dlayer with\n", - "data_format=\"channels_first\",\n", - " setaxis=1inBatchNormalization. \n",
- " - momentum: Momentum for the moving average. \n", - "
- epsilon: Small float added to variance to avoid dividing by zero. \n", - "
- center: If True, add offset of
betato normalized tensor.\n", - " If False,betais ignored. \n",
- " - scale: If True, multiply by
gamma.\n", - " If False,gammais not used.\n", - " When the next layer is linear (also e.g.nn.relu),\n", - " this can be disabled since the scaling\n", - " will be done by the next layer. \n",
- " - beta_initializer: Initializer for the beta weight. \n", - "
- gamma_initializer: Initializer for the gamma weight. \n", - "
- moving_mean_initializer: Initializer for the moving mean. \n", - "
- moving_variance_initializer: Initializer for the moving variance. \n", - "
- beta_regularizer: Optional regularizer for the beta weight. \n", - "
- gamma_regularizer: Optional regularizer for the gamma weight. \n", - "
- beta_constraint: Optional constraint for the beta weight. \n", - "
- gamma_constraint: Optional constraint for the gamma weight. \n", - "
![](\"imgs/vgg16.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# VGG19"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "![](\"imgs/vgg19.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "source": [
- "# `keras.applications`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "subslide"
- }
- },
- "outputs": [
- {
- "name": "stderr",
- "output_type": "stream",
- "text": [
- "Using TensorFlow backend.\n"
- ]
- }
- ],
- "source": [
- "from keras.applications import VGG16\n",
- "from keras.applications.imagenet_utils import preprocess_input, decode_predictions\n",
- "import os"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "# -- Jupyter/IPython way to see documentation\n",
- "# please focus on parameters (e.g. include top)\n",
- "VGG16??"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5\n"
- ]
- }
- ],
- "source": [
- "vgg16 = VGG16(include_top=True, weights='imagenet')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "If you're wondering **where** this `HDF5` files with weights is stored, please take a look at `~/.keras/models/`"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### HandsOn VGG16 - Pre-trained Weights"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "IMAGENET_FOLDER = 'imgs/imagenet' #in the repo"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[31mapricot_565.jpeg\u001b[m\u001b[m \u001b[31mapricot_787.jpeg\u001b[m\u001b[m \u001b[31mstrawberry_1174.jpeg\u001b[m\u001b[m\r\n",
- "\u001b[31mapricot_696.jpeg\u001b[m\u001b[m \u001b[31mstrawberry_1157.jpeg\u001b[m\u001b[m \u001b[31mstrawberry_1189.jpeg\u001b[m\u001b[m\r\n"
- ]
- }
- ],
- "source": [
- "!ls imgs/imagenet"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "![](\"imgs/imagenet/strawberry_1157.jpeg\")
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "skip"
- }
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Input image shape: (1, 224, 224, 3)\n",
- "Downloading data from https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json\n",
- "Predicted: [[('n07745940', 'strawberry', 0.98483676), ('n07836838', 'chocolate_sauce', 0.0073711565), ('n07614500', 'ice_cream', 0.0030998574), ('n04332243', 'strainer', 0.0025101686), ('n04476259', 'tray', 0.00060175249)]]\n"
- ]
- }
- ],
- "source": [
- "from keras.preprocessing import image\n",
- "import numpy as np\n",
- "\n",
- "img_path = os.path.join(IMAGENET_FOLDER, 'strawberry_1157.jpeg')\n",
- "img = image.load_img(img_path, target_size=(224, 224))\n",
- "x = image.img_to_array(img)\n",
- "x = np.expand_dims(x, axis=0)\n",
- "x = preprocess_input(x)\n",
- "print('Input image shape:', x.shape)\n",
- "\n",
- "preds = vgg16.predict(x)\n",
- "print('Predicted:', decode_predictions(preds))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "![](\"imgs/imagenet/apricot_696.jpeg\")
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Input image shape: (1, 224, 224, 3)\n",
- "Predicted: [[('n07747607', 'orange', 0.87526792), ('n07749582', 'lemon', 0.03620464), ('n07717556', 'butternut_squash', 0.021843448), ('n03937543', 'pill_bottle', 0.0126132), ('n03942813', 'ping-pong_ball', 0.0054204506)]]\n"
- ]
- }
- ],
- "source": [
- "img_path = os.path.join(IMAGENET_FOLDER, 'apricot_696.jpeg')\n",
- "img = image.load_img(img_path, target_size=(224, 224))\n",
- "x = image.img_to_array(img)\n",
- "x = np.expand_dims(x, axis=0)\n",
- "x = preprocess_input(x)\n",
- "print('Input image shape:', x.shape)\n",
- "\n",
- "preds = vgg16.predict(x)\n",
- "print('Predicted:', decode_predictions(preds))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true
- },
- "source": [
- "![](\"imgs/imagenet/apricot_565.jpeg\")
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Input image shape: (1, 224, 224, 3)\n",
- "Predicted: [[('n07718472', 'cucumber', 0.29338178), ('n07716358', 'zucchini', 0.2383192), ('n04596742', 'wok', 0.042132568), ('n07716906', 'spaghetti_squash', 0.038422), ('n07711569', 'mashed_potato', 0.036552209)]]\n"
- ]
- }
- ],
- "source": [
- "img_path = os.path.join(IMAGENET_FOLDER, 'apricot_565.jpeg')\n",
- "img = image.load_img(img_path, target_size=(224, 224))\n",
- "x = image.img_to_array(img)\n",
- "x = np.expand_dims(x, axis=0)\n",
- "x = preprocess_input(x)\n",
- "print('Input image shape:', x.shape)\n",
- "\n",
- "preds = vgg16.predict(x)\n",
- "print('Predicted:', decode_predictions(preds))"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "deletable": true,
- "editable": true,
- "slideshow": {
- "slide_type": "slide"
- }
- },
- "source": [
- "# Hands On:\n",
- "\n",
- "### Try to do the same with VGG19 Model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true,
- "deletable": true,
- "editable": true
- },
- "outputs": [],
- "source": [
- "# from keras.applications import VGG19"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Residual Networks"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "![](\"imgs/resnet_bb.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## ResNet 50"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "![](\"imgs/resnet34.png\")
"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "## from keras.applications import ..."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.5.2"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 0
-}
diff --git a/2.5 HyperParameter Tuning.ipynb b/2.5 HyperParameter Tuning.ipynb
deleted file mode 100644
index 9a52493..0000000
--- a/2.5 HyperParameter Tuning.ipynb
+++ /dev/null
@@ -1,268 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# HyperParameter Tuning"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### `keras.wrappers.scikit_learn`\n",
- "\n",
- "Example adapted from: [https://github.com/fchollet/keras/blob/master/examples/mnist_sklearn_wrapper.py]()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Problem: \n",
- "Builds simple CNN models on MNIST and uses sklearn's GridSearchCV to find best model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "import numpy as np\n",
- "np.random.seed(1337) # for reproducibility"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "from keras.datasets import mnist\n",
- "from keras.models import Sequential\n",
- "from keras.layers import Dense, Dropout, Activation, Flatten\n",
- "from keras.layers import Conv2D, MaxPooling2D\n",
- "from keras.utils import np_utils\n",
- "from keras.wrappers.scikit_learn import KerasClassifier\n",
- "from keras import backend as K"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "from sklearn.model_selection import GridSearchCV"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Data Preparation"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "nb_classes = 10\n",
- "\n",
- "# input image dimensions\n",
- "img_rows, img_cols = 28, 28"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "# load training data and do basic data normalization\n",
- "(X_train, y_train), (X_test, y_test) = mnist.load_data()\n",
- "\n",
- "if K.image_dim_ordering() == 'th':\n",
- " X_train = X_train.reshape(X_train.shape[0], 1, img_rows, img_cols)\n",
- " X_test = X_test.reshape(X_test.shape[0], 1, img_rows, img_cols)\n",
- " input_shape = (1, img_rows, img_cols)\n",
- "else:\n",
- " X_train = X_train.reshape(X_train.shape[0], img_rows, img_cols, 1)\n",
- " X_test = X_test.reshape(X_test.shape[0], img_rows, img_cols, 1)\n",
- " input_shape = (img_rows, img_cols, 1)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "X_train = X_train.astype('float32')\n",
- "X_test = X_test.astype('float32')\n",
- "X_train /= 255\n",
- "X_test /= 255\n",
- "\n",
- "# convert class vectors to binary class matrices\n",
- "y_train = np_utils.to_categorical(y_train, nb_classes)\n",
- "y_test = np_utils.to_categorical(y_test, nb_classes)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Build Model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "def make_model(dense_layer_sizes, nb_filters, nb_conv, nb_pool):\n",
- " '''Creates model comprised of 2 convolutional layers followed by dense layers\n",
- "\n",
- " dense_layer_sizes: List of layer sizes. This list has one number for each layer\n",
- " nb_filters: Number of convolutional filters in each convolutional layer\n",
- " nb_conv: Convolutional kernel size\n",
- " nb_pool: Size of pooling area for max pooling\n",
- " '''\n",
- "\n",
- " model = Sequential()\n",
- "\n",
- " model.add(Conv2D(nb_filters, (nb_conv, nb_conv),\n",
- " padding='valid', input_shape=input_shape))\n",
- " model.add(Activation('relu'))\n",
- " model.add(Conv2D(nb_filters, (nb_conv, nb_conv)))\n",
- " model.add(Activation('relu'))\n",
- " model.add(MaxPooling2D(pool_size=(nb_pool, nb_pool)))\n",
- " model.add(Dropout(0.25))\n",
- "\n",
- " model.add(Flatten())\n",
- " for layer_size in dense_layer_sizes:\n",
- " model.add(Dense(layer_size))\n",
- " model.add(Activation('relu'))\n",
- " model.add(Dropout(0.5))\n",
- " model.add(Dense(nb_classes))\n",
- " model.add(Activation('softmax'))\n",
- "\n",
- " model.compile(loss='categorical_crossentropy',\n",
- " optimizer='adadelta',\n",
- " metrics=['accuracy'])\n",
- "\n",
- " return model"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": [
- "dense_size_candidates = [[32], [64], [32, 32], [64, 64]]\n",
- "my_classifier = KerasClassifier(make_model, batch_size=32)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## GridSearch HyperParameters"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": false,
- "scrolled": false
- },
- "outputs": [],
- "source": [
- "validator = GridSearchCV(my_classifier,\n",
- " param_grid={'dense_layer_sizes': dense_size_candidates,\n",
- " # nb_epoch is avail for tuning even when not\n",
- " # an argument to model building function\n",
- " 'nb_epoch': [3, 6],\n",
- " 'nb_filters': [8],\n",
- " 'nb_conv': [3],\n",
- " 'nb_pool': [2]},\n",
- " scoring='neg_log_loss',\n",
- " n_jobs=1)\n",
- "validator.fit(X_train, y_train)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": false
- },
- "outputs": [],
- "source": [
- "print('The parameters of the best model are: ')\n",
- "print(validator.best_params_)\n",
- "\n",
- "# validator.best_estimator_ returns sklearn-wrapped version of best model.\n",
- "# validator.best_estimator_.model returns the (unwrapped) keras model\n",
- "best_model = validator.best_estimator_.model\n",
- "metric_names = best_model.metrics_names\n",
- "metric_values = best_model.evaluate(X_test, y_test)\n",
- "for metric, value in zip(metric_names, metric_values):\n",
- " print(metric, ': ', value)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "collapsed": true
- },
- "outputs": [],
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.5.2"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}
diff --git a/2.2 CNN HandsOn - MNIST Dataset.ipynb b/3. Fully Connected Networks and Embeddings/3.0 - MNIST Dataset.ipynb
similarity index 81%
rename from 2.2 CNN HandsOn - MNIST Dataset.ipynb
rename to 3. Fully Connected Networks and Embeddings/3.0 - MNIST Dataset.ipynb
index 4c4fbc6..d27e35e 100644
--- a/2.2 CNN HandsOn - MNIST Dataset.ipynb
+++ b/3. Fully Connected Networks and Embeddings/3.0 - MNIST Dataset.ipynb
@@ -3,21 +3,30 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
},
"source": [
- "# CNN HandsOn with Keras"
+ "# MNIST Dataset"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Also known as `digits` if you're familiar with `sklearn`:\n",
+ "\n",
+ "```python\n",
+ "\n",
+ "from sklearn.datasets import digits\n",
+ "\n",
+ "```"
]
},
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -26,14 +35,12 @@
"## Problem Definition\n",
"\n",
"*Recognize handwritten digits*\n",
- ""
+ ""
]
},
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -54,8 +61,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -72,9 +77,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -91,9 +94,7 @@
"execution_count": null,
"metadata": {
"code_folding": [],
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "fragment"
}
@@ -107,8 +108,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -121,9 +120,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -137,9 +134,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -153,9 +148,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -169,9 +162,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -185,9 +176,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -201,9 +190,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -217,9 +204,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -233,9 +218,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -249,9 +232,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true,
+ "collapsed": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -265,8 +246,6 @@
{
"cell_type": "markdown",
"metadata": {
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "slide"
}
@@ -284,8 +263,6 @@
"execution_count": null,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -302,8 +279,6 @@
"execution_count": null,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -317,9 +292,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": [
@@ -337,8 +310,6 @@
"execution_count": null,
"metadata": {
"collapsed": true,
- "deletable": true,
- "editable": true,
"slideshow": {
"slide_type": "subslide"
}
@@ -352,9 +323,7 @@
"cell_type": "code",
"execution_count": null,
"metadata": {
- "collapsed": false,
- "deletable": true,
- "editable": true
+ "collapsed": true
},
"outputs": [],
"source": []
@@ -376,9 +345,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.5.2"
+ "version": "3.5.3"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 1
}
diff --git a/3. Fully Connected Networks and Embeddings/3.1 Hidden Layer Representation and Embeddings.ipynb b/3. Fully Connected Networks and Embeddings/3.1 Hidden Layer Representation and Embeddings.ipynb
new file mode 100644
index 0000000..7eb1514
--- /dev/null
+++ b/3. Fully Connected Networks and Embeddings/3.1 Hidden Layer Representation and Embeddings.ipynb
@@ -0,0 +1,1495 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Fully Connected Feed-Forward Network\n",
+ "\n",
+ "In this notebook we will play with Feed-Forward FC-NN (Fully Connected Neural Network) for a *classification task*: \n",
+ "\n",
+ "Image Classification on MNIST Dataset\n",
+ "\n",
+ "**RECALL**\n",
+ "\n",
+ "In the FC-NN, the output of each layer is computed using the activations from the previous one, as follows:\n",
+ "\n",
+ "$$h_{i} = \\sigma(W_i h_{i-1} + b_i)$$\n",
+ "\n",
+ "where ${h}_i$ is the activation vector from the $i$-th layer (or the input data for $i=0$), ${W}_i$ and ${b}_i$ are the weight matrix and the bias vector for the $i$-th layer, respectively. \n",
+ "\\n\"+\n",
+ " \"
\"}};\n",
+ "\n",
+ " function display_loaded() {\n",
+ " if (window.Bokeh !== undefined) {\n",
+ " var el = document.getElementById(\"0af86eff-6a55-4644-ab84-9a6f5fcbeb3e\");\n",
+ " el.textContent = \"BokehJS \" + Bokeh.version + \" successfully loaded.\";\n",
+ " } else if (Date.now() < window._bokeh_timeout) {\n",
+ " setTimeout(display_loaded, 100)\n",
+ " }\n",
+ " }\n",
+ "\n",
+ " function run_callbacks() {\n",
+ " window._bokeh_onload_callbacks.forEach(function(callback) { callback() });\n",
+ " delete window._bokeh_onload_callbacks\n",
+ " console.info(\"Bokeh: all callbacks have finished\");\n",
+ " }\n",
+ "\n",
+ " function load_libs(js_urls, callback) {\n",
+ " window._bokeh_onload_callbacks.push(callback);\n",
+ " if (window._bokeh_is_loading > 0) {\n",
+ " console.log(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n",
+ " return null;\n",
+ " }\n",
+ " if (js_urls == null || js_urls.length === 0) {\n",
+ " run_callbacks();\n",
+ " return null;\n",
+ " }\n",
+ " console.log(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n",
+ " window._bokeh_is_loading = js_urls.length;\n",
+ " for (var i = 0; i < js_urls.length; i++) {\n",
+ " var url = js_urls[i];\n",
+ " var s = document.createElement('script');\n",
+ " s.src = url;\n",
+ " s.async = false;\n",
+ " s.onreadystatechange = s.onload = function() {\n",
+ " window._bokeh_is_loading--;\n",
+ " if (window._bokeh_is_loading === 0) {\n",
+ " console.log(\"Bokeh: all BokehJS libraries loaded\");\n",
+ " run_callbacks()\n",
+ " }\n",
+ " };\n",
+ " s.onerror = function() {\n",
+ " console.warn(\"failed to load library \" + url);\n",
+ " };\n",
+ " console.log(\"Bokeh: injecting script tag for BokehJS library: \", url);\n",
+ " document.getElementsByTagName(\"head\")[0].appendChild(s);\n",
+ " }\n",
+ " };var element = document.getElementById(\"0af86eff-6a55-4644-ab84-9a6f5fcbeb3e\");\n",
+ " if (element == null) {\n",
+ " console.log(\"Bokeh: ERROR: autoload.js configured with elementid '0af86eff-6a55-4644-ab84-9a6f5fcbeb3e' but no matching script tag was found. \")\n",
+ " return false;\n",
+ " }\n",
+ "\n",
+ " var js_urls = [\"https://cdn.pydata.org/bokeh/release/bokeh-0.12.5.min.js\", \"https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.5.min.js\"];\n",
+ "\n",
+ " var inline_js = [\n",
+ " function(Bokeh) {\n",
+ " Bokeh.set_log_level(\"info\");\n",
+ " },\n",
+ " \n",
+ " function(Bokeh) {\n",
+ " \n",
+ " },\n",
+ " \n",
+ " function(Bokeh) {\n",
+ " \n",
+ " document.getElementById(\"0af86eff-6a55-4644-ab84-9a6f5fcbeb3e\").textContent = \"BokehJS is loading...\";\n",
+ " },\n",
+ " function(Bokeh) {\n",
+ " console.log(\"Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-0.12.5.min.css\");\n",
+ " Bokeh.embed.inject_css(\"https://cdn.pydata.org/bokeh/release/bokeh-0.12.5.min.css\");\n",
+ " console.log(\"Bokeh: injecting CSS: https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.5.min.css\");\n",
+ " Bokeh.embed.inject_css(\"https://cdn.pydata.org/bokeh/release/bokeh-widgets-0.12.5.min.css\");\n",
+ " }\n",
+ " ];\n",
+ "\n",
+ " function run_inline_js() {\n",
+ " \n",
+ " if ((window.Bokeh !== undefined) || (force === true)) {\n",
+ " for (var i = 0; i < inline_js.length; i++) {\n",
+ " inline_js[i](window.Bokeh);\n",
+ " }if (force === true) {\n",
+ " display_loaded();\n",
+ " }} else if (Date.now() < window._bokeh_timeout) {\n",
+ " setTimeout(run_inline_js, 100);\n",
+ " } else if (!window._bokeh_failed_load) {\n",
+ " console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n",
+ " window._bokeh_failed_load = true;\n",
+ " } else if (force !== true) {\n",
+ " var cell = $(document.getElementById(\"0af86eff-6a55-4644-ab84-9a6f5fcbeb3e\")).parents('.cell').data().cell;\n",
+ " cell.output_area.append_execute_result(NB_LOAD_WARNING)\n",
+ " }\n",
+ "\n",
+ " }\n",
+ "\n",
+ " if (window._bokeh_is_loading === 0) {\n",
+ " console.log(\"Bokeh: BokehJS loaded, going straight to plotting\");\n",
+ " run_inline_js();\n",
+ " } else {\n",
+ " load_libs(js_urls, function() {\n",
+ " console.log(\"Bokeh: BokehJS plotting callback run at\", now());\n",
+ " run_inline_js();\n",
+ " });\n",
+ " }\n",
+ "}(this));"
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "from bokeh.plotting import figure, output_notebook, show\n",
+ "\n",
+ "output_notebook()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 74,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/html": [
+ "\n",
+ "\n",
+ " \\n\"+\n", + " \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n", + " \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n", + " \"
\\n\"+\n", + " \"- \\n\"+\n",
+ " \"
- re-rerun `output_notebook()` to attempt to load from CDN again, or \\n\"+\n", + " \"
- use INLINE resources instead, as so: \\n\"+\n", + " \"
\\n\"+\n",
+ " \"from bokeh.resources import INLINE\\n\"+\n",
+ " \"output_notebook(resources=INLINE)\\n\"+\n",
+ " \"\\n\"+\n",
+ " \"\n",
+ " \n",
+ "
\n",
+ ""
+ ]
+ },
+ "metadata": {},
+ "output_type": "display_data"
+ }
+ ],
+ "source": [
+ "p = figure(plot_width=600, plot_height=600)\n",
+ "\n",
+ "colors = [x for x in 'blue-green-red-cyan-magenta-yellow-black-purple-coral-lime'.split('-')]\n",
+ "colors_map = colors_map[:1000]\n",
+ "for cl in range(nb_classes):\n",
+ " indices = np.where(colors_map==cl)\n",
+ " p.circle(X_tsne[indices, 0].ravel(), X_tsne[indices, 1].ravel(), size=7, \n",
+ " color=colors[cl], alpha=0.4, legend=str(cl))\n",
+ "\n",
+ "# show the results\n",
+ "p.legend.location = 'bottom_right'\n",
+ "show(p)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Note: We used `default` TSNE parameters. Better results can be achieved by tuning TSNE Hyper-parameters"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Exercise 1: \n",
+ "\n",
+ "### Try with a different algorithm to create the manifold"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 75,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "from sklearn.manifold import MDS"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "## Your code here"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Exercise 2: \n",
+ "\n",
+ "### Try extracting the Hidden features of the First and the Last layer of the model"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "## Your code here"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": [
+ "## Try using the `get_activations` function relying on keras backend\n",
+ "def get_activations(model, layer, X_batch):\n",
+ " activations_f = K.function([model.layers[0].input, K.learning_phase()], [layer.output,])\n",
+ " activations = activations_f((X_batch, False))\n",
+ " return activations"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {
+ "collapsed": true
+ },
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.5.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}
diff --git a/3.1 AutoEncoders and Embeddings.ipynb b/3.1 AutoEncoders and Embeddings.ipynb
deleted file mode 100644
index a14b895..0000000
--- a/3.1 AutoEncoders and Embeddings.ipynb
+++ /dev/null
@@ -1,2250 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Unsupervised learning"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### AutoEncoders "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "An autoencoder, is an artificial neural network used for learning efficient codings. \n",
- "\n",
- "The aim of an autoencoder is to learn a representation (encoding) for a set of data, typically for the purpose of dimensionality reduction. "
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
- "source": [
- "![](\"imgs/autoencoder.png\")
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Unsupervised learning is a type of machine learning algorithm used to draw inferences from datasets consisting of input data without labeled responses. The most common unsupervised learning method is cluster analysis, which is used for exploratory data analysis to find hidden patterns or grouping in data."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {
- "collapsed": false,
- "scrolled": false
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Train on 60000 samples, validate on 10000 samples\n",
- "Epoch 1/50\n",
- "60000/60000 [==============================] - 20s - loss: 0.3832 - val_loss: 0.2730\n",
- "Epoch 2/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.2660 - val_loss: 0.2557\n",
- "Epoch 3/50\n",
- "60000/60000 [==============================] - 18s - loss: 0.2455 - val_loss: 0.2331\n",
- "Epoch 4/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.2254 - val_loss: 0.2152\n",
- "Epoch 5/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.2099 - val_loss: 0.2018\n",
- "Epoch 6/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1982 - val_loss: 0.1917\n",
- "Epoch 7/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1893 - val_loss: 0.1840\n",
- "Epoch 8/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1824 - val_loss: 0.1778\n",
- "Epoch 9/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1766 - val_loss: 0.1725\n",
- "Epoch 10/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1715 - val_loss: 0.1676\n",
- "Epoch 11/50\n",
- "60000/60000 [==============================] - 18s - loss: 0.1669 - val_loss: 0.1632\n",
- "Epoch 12/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1626 - val_loss: 0.1593\n",
- "Epoch 13/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1587 - val_loss: 0.1554\n",
- "Epoch 14/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1551 - val_loss: 0.1520\n",
- "Epoch 15/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1519 - val_loss: 0.1489\n",
- "Epoch 16/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1489 - val_loss: 0.1461\n",
- "Epoch 17/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1461 - val_loss: 0.1435\n",
- "Epoch 18/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1435 - val_loss: 0.1408\n",
- "Epoch 19/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1410 - val_loss: 0.1385\n",
- "Epoch 20/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1387 - val_loss: 0.1362\n",
- "Epoch 21/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1365 - val_loss: 0.1340\n",
- "Epoch 22/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1344 - val_loss: 0.1320\n",
- "Epoch 23/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1324 - val_loss: 0.1301\n",
- "Epoch 24/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1305 - val_loss: 0.1282\n",
- "Epoch 25/50\n",
- "60000/60000 [==============================] - 18s - loss: 0.1287 - val_loss: 0.1265\n",
- "Epoch 26/50\n",
- "60000/60000 [==============================] - 18s - loss: 0.1270 - val_loss: 0.1248\n",
- "Epoch 27/50\n",
- "60000/60000 [==============================] - 18s - loss: 0.1254 - val_loss: 0.1232\n",
- "Epoch 28/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1238 - val_loss: 0.1217\n",
- "Epoch 29/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1224 - val_loss: 0.1203\n",
- "Epoch 30/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1210 - val_loss: 0.1189\n",
- "Epoch 31/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1197 - val_loss: 0.1176\n",
- "Epoch 32/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1184 - val_loss: 0.1163\n",
- "Epoch 33/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1173 - val_loss: 0.1152\n",
- "Epoch 34/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1161 - val_loss: 0.1141\n",
- "Epoch 35/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1151 - val_loss: 0.1131\n",
- "Epoch 36/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1141 - val_loss: 0.1122\n",
- "Epoch 37/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1132 - val_loss: 0.1112\n",
- "Epoch 38/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1123 - val_loss: 0.1104\n",
- "Epoch 39/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1115 - val_loss: 0.1095\n",
- "Epoch 40/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1107 - val_loss: 0.1088\n",
- "Epoch 41/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1100 - val_loss: 0.1081\n",
- "Epoch 42/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1093 - val_loss: 0.1074\n",
- "Epoch 43/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1087 - val_loss: 0.1068\n",
- "Epoch 44/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1081 - val_loss: 0.1062\n",
- "Epoch 45/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1075 - val_loss: 0.1057\n",
- "Epoch 46/50\n",
- "60000/60000 [==============================] - 19s - loss: 0.1070 - val_loss: 0.1052\n",
- "Epoch 47/50\n",
- "60000/60000 [==============================] - 20s - loss: 0.1065 - val_loss: 0.1047\n",
- "Epoch 48/50\n",
- "60000/60000 [==============================] - 17s - loss: 0.1061 - val_loss: 0.1043\n",
- "Epoch 49/50\n",
- "60000/60000 [==============================] - 29s - loss: 0.1056 - val_loss: 0.1039\n",
- "Epoch 50/50\n",
- "60000/60000 [==============================] - 21s - loss: 0.1052 - val_loss: 0.1034\n"
- ]
- },
- {
- "data": {
- "text/plain": [
- "