10. Reconsturcting Inputs using Autoencoders/10.11 Implementing Contractive Autoencoders.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise: Implementing Contractive Autoencoders"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Building the contractvie autoencoder is just as same as building the autoencoder except that here we use contractive loss. So instead of looking at the whole code we will only see how to implement the contractive loss. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know that contractive loss is sum of mean squared error and penalty term,\n",
    "\n",
    "$$ L = \\lVert X - \\hat{X} \\rVert_2^2  + \\lambda \\lVert J_f(X) \\rVert_F^2  $$\n",
    "\n",
    "Considering sigmoid as an activation function, the penalty term can be derived as:\n",
    "\n",
    "$$  L = \\lVert X - \\hat{X} \\rVert_2^2 + \\sum_j [h_j(1 - h_j)]^2 \\sum_i (W_{ji}^T)^2   $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contractive Loss Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def contractive_loss(y_pred, y_true):\n",
    "\n",
    "    lm = 1e-4\n",
    "\n",
    "    MSE = K.mean(K.square(y_true - y_pred), axis=1)\n",
    "\n",
    "    weights = K.variable(value=model.get_layer('encoder_layer').get_weights()[0]) \n",
    "    weights = K.transpose(weights) \n",
    "\n",
    "    h = model.get_layer('encoder_layer').output\n",
    "\n",
    "    penalty_term =  K.sum(((h * (1 - h))**2) * K.sum(weights**2, axis=1), axis=1)\n",
    "\n",
    "    Loss = MSE + (lm * penalty_term)\n",
    "\n",
    "\n",
    "    return Loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have learned how to implement contractive loss, try implementing contractive autoencoders with the above contractive loss as an exercise. It is essentially same as vanilla autoencoders given in 10.03 except replace the binary cross entropy with the contractive loss while compiling the model. In the next section, we will learn how to use Variational autoencoders (VAE) for generating images. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}