experiments/text_generation_shakespeare_rnn/text_generation_shakespeare_rnn.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JL3ZPJPgC4wC"
   },
   "source": [
    "# Shakespeare Text Generation (using RNN LSTM)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eoWGtkL8PPJ3"
   },
   "source": [
    "> - 🤖 See [full list of Machine Learning Experiments](https://github.com/trekhleb/machine-learning-experiments) on **GitHub**<br/><br/>\n",
    "> - ▶️ **Interactive Demo**: [try this model and other machine learning experiments in action](https://trekhleb.github.io/machine-learning-experiments/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bui0MyTjv1Mp"
   },
   "source": [
    "## Experiment overview\n",
    "\n",
    "In this experiment we will use character-based [Recurrent Neural Network](https://en.wikipedia.org/wiki/Recurrent_neural_network) (RNN) to generate a Shakespeare's-like text based on the Shakespeare dataset from [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) blog post.\n",
    "\n",
    "For this experiment we will use [Tensorflow v2](https://www.tensorflow.org/) with its [Keras API](https://www.tensorflow.org/guide/keras).\n",
    "\n",
    "![text_generation_shakespeare_rnn.jpg](../../demos/src/images/text_generation_shakespeare_rnn.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3XxEZuRNIzHH"
   },
   "source": [
    "_Inspired by [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IDJctDhhEDTD"
   },
   "source": [
    "## Import dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1347,
     "status": "ok",
     "timestamp": 1584519798294,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Z2xZnxncD2Ep",
    "outputId": "9e110fb9-a6d1-4bd6-818c-2198114e3133"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow 2.x selected.\n"
     ]
    }
   ],
   "source": [
    "# Selecting Tensorflow version v2 (the command is relevant for Colab only).\n",
    "# %tensorflow_version 2.x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2811,
     "status": "ok",
     "timestamp": 1584519800384,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "SpueB6zADYgE",
    "outputId": "3f4e9997-8ce8-4ef8-f8f9-320c9a44e03b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python version: 3.6.9\n",
      "Tensorflow version: 2.1.0\n",
      "Keras version: 2.2.4-tf\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import platform\n",
    "import time\n",
    "import pathlib\n",
    "import os\n",
    "\n",
    "print('Python version:', platform.python_version())\n",
    "print('Tensorflow version:', tf.__version__)\n",
    "print('Keras version:', tf.keras.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ciI_JnnNEGCw"
   },
   "source": [
    "## Download the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2533,
     "status": "ok",
     "timestamp": 1584519800396,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Vk2HldshEOph",
    "outputId": "45342704-eb1c-4863-a4e4-8a94d9cc764b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt\n",
      "1122304/1115394 [==============================] - 0s 0us/step\n",
      "/content/tmp/datasets/shakespeare.txt\n"
     ]
    }
   ],
   "source": [
    "cache_dir = './tmp'\n",
    "dataset_file_name = 'shakespeare.txt'\n",
    "dataset_file_origin = 'https://storage.googleapis.com/download.tensorflow.org/data/shakespeare.txt'\n",
    "\n",
    "dataset_file_path = tf.keras.utils.get_file(\n",
    "    fname=dataset_file_name,\n",
    "    origin=dataset_file_origin,\n",
    "    cache_dir=pathlib.Path(cache_dir).absolute()\n",
    ")\n",
    "\n",
    "print(dataset_file_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eKTy6YS5Gx-g"
   },
   "source": [
    "## Analyze the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2199,
     "status": "ok",
     "timestamp": 1584519800398,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "CzfX7hK8G0bn",
    "outputId": "dbcd9c2e-967d-404f-ebd4-387ee0b629d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of text: 1115394 characters\n"
     ]
    }
   ],
   "source": [
    "# Reading the database file.\n",
    "text = open(dataset_file_path, mode='r').read()\n",
    "\n",
    "print('Length of text: {} characters'.format(len(text)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1774,
     "status": "ok",
     "timestamp": 1584519800402,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "nknOAOuoH-Av",
    "outputId": "7030d3f8-c6a8-43e0-81a0-c1e87d677314"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First Citizen:\n",
      "Before we proceed any further, hear me speak.\n",
      "\n",
      "All:\n",
      "Speak, speak.\n",
      "\n",
      "First Citizen:\n",
      "You are all resolved rather to die than to famish?\n",
      "\n",
      "All:\n",
      "Resolved. resolved.\n",
      "\n",
      "First Citizen:\n",
      "First, you know Caius Marcius is chief enemy to the people.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Take a look at the first 250 characters in text.\n",
    "print(text[:250])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1590,
     "status": "ok",
     "timestamp": 1584519800403,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "VPXkB5bOITqi",
    "outputId": "69fdb374-4b35-442d-a934-7017245ea18e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65 unique characters\n",
      "vocab: ['\\n', ' ', '!', '$', '&', \"'\", ',', '-', '.', '3', ':', ';', '?', 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']\n"
     ]
    }
   ],
   "source": [
    "# The unique characters in the file\n",
    "vocab = sorted(set(text))\n",
    "\n",
    "print('{} unique characters'.format(len(vocab)))\n",
    "print('vocab:', vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GqpuKh9HMNnf"
   },
   "source": [
    "## Process the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "6dj4e-AGMaV4"
   },
   "source": [
    "### Vectorize the text\n",
    "\n",
    "Before feeding the text to our RNN we need to convert the text from a sequence of characters to a sequence of numbers. To do so we will detect all unique characters in the text, form a vocabulary out of it and replace each character with its index in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1510,
     "status": "ok",
     "timestamp": 1584519800417,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "xFFpuXfGMPq2",
    "outputId": "530395b0-62e2-41e5-c0bf-63b2ca9da7cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  '\\n':   0,\n",
      "  ' ' :   1,\n",
      "  '!' :   2,\n",
      "  '$' :   3,\n",
      "  '&' :   4,\n",
      "  \"'\" :   5,\n",
      "  ',' :   6,\n",
      "  '-' :   7,\n",
      "  '.' :   8,\n",
      "  '3' :   9,\n",
      "  ':' :  10,\n",
      "  ';' :  11,\n",
      "  '?' :  12,\n",
      "  'A' :  13,\n",
      "  'B' :  14,\n",
      "  'C' :  15,\n",
      "  'D' :  16,\n",
      "  'E' :  17,\n",
      "  'F' :  18,\n",
      "  'G' :  19,\n",
      "  ...\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "# Map characters to their indices in vocabulary.\n",
    "char2index = {char: index for index, char in enumerate(vocab)}\n",
    "\n",
    "print('{')\n",
    "for char, _ in zip(char2index, range(20)):\n",
    "    print('  {:4s}: {:3d},'.format(repr(char), char2index[char]))\n",
    "print('  ...\\n}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1466,
     "status": "ok",
     "timestamp": 1584519800417,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "lQB33zI7NkRo",
    "outputId": "ef5bc93f-7bad-437f-f70a-9eb53d895613"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['\\n' ' ' '!' '$' '&' \"'\" ',' '-' '.' '3' ':' ';' '?' 'A' 'B' 'C' 'D' 'E'\n",
      " 'F' 'G' 'H' 'I' 'J' 'K' 'L' 'M' 'N' 'O' 'P' 'Q' 'R' 'S' 'T' 'U' 'V' 'W'\n",
      " 'X' 'Y' 'Z' 'a' 'b' 'c' 'd' 'e' 'f' 'g' 'h' 'i' 'j' 'k' 'l' 'm' 'n' 'o'\n",
      " 'p' 'q' 'r' 's' 't' 'u' 'v' 'w' 'x' 'y' 'z']\n"
     ]
    }
   ],
   "source": [
    "# Map character indices to characters from vacabulary.\n",
    "index2char = np.array(vocab)\n",
    "print(index2char)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1367,
     "status": "ok",
     "timestamp": 1584519800418,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "DXUAlYmvN_Rj",
    "outputId": "ef3412c4-68e0-4362-e093-feba628b7079"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "text_as_int length: 1115394\n",
      "'First Citizen:\\n' --> array([18, 47, 56, 57, 58,  1, 15, 47, 58, 47, 64, 43, 52, 10,  0])\n"
     ]
    }
   ],
   "source": [
    "# Convert chars in text to indices.\n",
    "text_as_int = np.array([char2index[char] for char in text])\n",
    "\n",
    "print('text_as_int length: {}'.format(len(text_as_int)))\n",
    "print('{} --> {}'.format(repr(text[:15]), repr(text_as_int[:15])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CHv5HhUuTQYS"
   },
   "source": [
    "## Create training sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1227,
     "status": "ok",
     "timestamp": 1584519800418,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "9tHT8IXGTTtZ",
    "outputId": "489b7e07-e898-49bc-e099-a9e7c4791c8c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "examples_per_epoch: 11043\n"
     ]
    }
   ],
   "source": [
    "# The maximum length sentence we want for a single input in characters.\n",
    "sequence_length = 100\n",
    "examples_per_epoch = len(text) // (sequence_length + 1)\n",
    "\n",
    "print('examples_per_epoch:', examples_per_epoch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1331,
     "status": "ok",
     "timestamp": 1584519800779,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Pf5tMMDnUP3q",
    "outputId": "f8e191f6-df0b-426b-9905-af62be4d605c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F\n",
      "i\n",
      "r\n",
      "s\n",
      "t\n"
     ]
    }
   ],
   "source": [
    "# Create training dataset.\n",
    "char_dataset = tf.data.Dataset.from_tensor_slices(text_as_int)\n",
    "\n",
    "for char in char_dataset.take(5):\n",
    "    print(index2char[char.numpy()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3082,
     "status": "ok",
     "timestamp": 1584519802697,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Ap71VjB2Vuct",
    "outputId": "0c05de4a-d66c-4f20-83a7-e584d853e98d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sequences count: 11043\n",
      "\n",
      "'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou '\n",
      "'are all resolved rather to die than to famish?\\n\\nAll:\\nResolved. resolved.\\n\\nFirst Citizen:\\nFirst, you k'\n",
      "\"now Caius Marcius is chief enemy to the people.\\n\\nAll:\\nWe know't, we know't.\\n\\nFirst Citizen:\\nLet us ki\"\n",
      "\"ll him, and we'll have corn at our own price.\\nIs't a verdict?\\n\\nAll:\\nNo more talking on't; let it be d\"\n",
      "'one: away, away!\\n\\nSecond Citizen:\\nOne word, good citizens.\\n\\nFirst Citizen:\\nWe are accounted poor citi'\n"
     ]
    }
   ],
   "source": [
    "# Generate batched sequences out of the char_dataset.\n",
    "sequences = char_dataset.batch(sequence_length + 1, drop_remainder=True)\n",
    "\n",
    "# Sequences size is the same as examples_per_epoch.\n",
    "print('Sequences count: {}'.format(len(list(sequences.as_numpy_iterator()))));\n",
    "print()\n",
    "\n",
    "# Sequences examples.\n",
    "for item in sequences.take(5):\n",
    "    print(repr(''.join(index2char[item.numpy()])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y8spPCfe-iTn"
   },
   "outputs": [],
   "source": [
    "# sequences shape:\n",
    "# - 11043 sequences\n",
    "# - Each sequence of length 101\n",
    "#\n",
    "#\n",
    "#    101     101          101\n",
    "# [(.....) (.....) ...  (.....)]\n",
    "#\n",
    "# <---------- 11043 ----------->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HdcrcUs4Xxso"
   },
   "source": [
    "For each sequence, duplicate and shift it to form the input and target text. For example, say `sequence_length` is `4` and our text is `Hello`. The input sequence would be `Hell`, and the target sequence `ello`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9fxvXsP0XFDh"
   },
   "outputs": [],
   "source": [
    "def split_input_target(chunk):\n",
    "    input_text = chunk[:-1]\n",
    "    target_text = chunk[1:]\n",
    "    return input_text, target_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6915,
     "status": "ok",
     "timestamp": 1584519807164,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "454rWIQYXXRY",
    "outputId": "de3a70e1-49e8-4f68-d988-77d1856a2e60"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dataset size: 11043\n"
     ]
    }
   ],
   "source": [
    "dataset = sequences.map(split_input_target)\n",
    "\n",
    "# Dataset size is the same as examples_per_epoch.\n",
    "# But each element of a sequence is now has length of `sequence_length`\n",
    "# and not `sequence_length + 1`.\n",
    "print('dataset size: {}'.format(len(list(dataset.as_numpy_iterator()))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6548,
     "status": "ok",
     "timestamp": 1584519807165,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Kuoh4tCdYCck",
    "outputId": "b03368eb-4c92-4662-95fe-d64f6bec308d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input sequence size: 100\n",
      "Target sequence size: 100\n",
      "\n",
      "Input: 'First Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou'\n",
      "Target: 'irst Citizen:\\nBefore we proceed any further, hear me speak.\\n\\nAll:\\nSpeak, speak.\\n\\nFirst Citizen:\\nYou '\n"
     ]
    }
   ],
   "source": [
    "for input_example, target_example in dataset.take(1):\n",
    "    print('Input sequence size:', repr(len(input_example.numpy())))\n",
    "    print('Target sequence size:', repr(len(target_example.numpy())))\n",
    "    print()\n",
    "    print('Input:', repr(''.join(index2char[input_example.numpy()])))\n",
    "    print('Target:', repr(''.join(index2char[target_example.numpy()])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "cp0tl0sN807l"
   },
   "outputs": [],
   "source": [
    "# dataset shape:\n",
    "# - 11043 sequences\n",
    "# - Each sequence is a tuple of 2 sub-sequences of length 100 (input_text and target_text)\n",
    "#\n",
    "#\n",
    "#    100       100           100\n",
    "# /(.....)\\ /(.....)\\ ... /(.....)\\  <-- input_text\n",
    "# \\(.....)/ \\(.....)/     \\(.....)/  <-- target_text\n",
    "#\n",
    "# <----------- 11043 ------------->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BDYHEJ0pY1ai"
   },
   "source": [
    "Each index of these vectors are processed as one time step. For the input at time step 0, the model receives the index for \"F\" and trys to predict the index for \"i\" as the next character. At the next timestep, it does the same thing but the RNN considers the previous step context in addition to the current input character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6469,
     "status": "ok",
     "timestamp": 1584519807584,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "C-0zpv53Y2o4",
    "outputId": "05fb1543-308c-42f9-d2ee-003c2454772b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Step  0\n",
      "  input: 18 ('F')\n",
      "  expected output: 47 ('i')\n",
      "Step  1\n",
      "  input: 47 ('i')\n",
      "  expected output: 56 ('r')\n",
      "Step  2\n",
      "  input: 56 ('r')\n",
      "  expected output: 57 ('s')\n",
      "Step  3\n",
      "  input: 57 ('s')\n",
      "  expected output: 58 ('t')\n",
      "Step  4\n",
      "  input: 58 ('t')\n",
      "  expected output: 1 (' ')\n"
     ]
    }
   ],
   "source": [
    "for i, (input_idx, target_idx) in enumerate(zip(input_example[:5], target_example[:5])):\n",
    "    print('Step {:2d}'.format(i))\n",
    "    print('  input: {} ({:s})'.format(input_idx, repr(index2char[input_idx])))\n",
    "    print('  expected output: {} ({:s})'.format(target_idx, repr(index2char[target_idx])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1iDlp40lC5YB"
   },
   "source": [
    "## Split training sequences into batches\n",
    "\n",
    "We used `tf.data` to split the text into manageable sequences. But before feeding this data into the model, we need to shuffle the data and pack it into batches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6256,
     "status": "ok",
     "timestamp": 1584519807585,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "eDq-wa5EC3wW",
    "outputId": "d13224e8-062a-4df8-c232-04f60d572e8d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BatchDataset shapes: ((64, 100), (64, 100)), types: (tf.int64, tf.int64)>"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Batch size.\n",
    "BATCH_SIZE = 64\n",
    "\n",
    "# Buffer size to shuffle the dataset (TF data is designed to work\n",
    "# with possibly infinite sequences, so it doesn't attempt to shuffle\n",
    "# the entire sequence in memory. Instead, it maintains a buffer in\n",
    "# which it shuffles elements).\n",
    "BUFFER_SIZE = 10000\n",
    "\n",
    "dataset = dataset.shuffle(BUFFER_SIZE).batch(BATCH_SIZE, drop_remainder=True)\n",
    "\n",
    "dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8229,
     "status": "ok",
     "timestamp": 1584519809742,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "1x4puZiiOlyl",
    "outputId": "1debcd12-46ba-4b5f-aa56-a162fbe76f1a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Batched dataset size: 172\n"
     ]
    }
   ],
   "source": [
    "print('Batched dataset size: {}'.format(len(list(dataset.as_numpy_iterator()))))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9749,
     "status": "ok",
     "timestamp": 1584519811500,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "b_kYgvQGBO0U",
    "outputId": "2dfda8dc-a501-4e5f-8835-6c87b8f77051"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1st batch: input_text: tf.Tensor(\n",
      "[[39 58 59 ... 57 58 53]\n",
      " [47 52  1 ...  1 63 53]\n",
      " [53 61  1 ... 46 57  6]\n",
      " ...\n",
      " [ 1 57 43 ... 52 42  8]\n",
      " [58 46 43 ... 43  1 58]\n",
      " [56 43 47 ... 53 59  1]], shape=(64, 100), dtype=int64)\n",
      "\n",
      "1st batch: target_text: tf.Tensor(\n",
      "[[58 59 56 ... 58 53 41]\n",
      " [52  1 63 ... 63 53 59]\n",
      " [61  1 50 ... 57  6  1]\n",
      " ...\n",
      " [57 43 43 ... 42  8  0]\n",
      " [46 43 43 ...  1 58 53]\n",
      " [43 47 45 ... 59  1 61]], shape=(64, 100), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "for input_text, target_text in dataset.take(1):\n",
    "    print('1st batch: input_text:', input_text)\n",
    "    print()\n",
    "    print('1st batch: target_text:', target_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UkDCH15v_2I6"
   },
   "outputs": [],
   "source": [
    "# dataset shape:\n",
    "# - 172 batches\n",
    "# - 64 sequences per batch\n",
    "# - Each sequence is a tuple of 2 sub-sequences of length 100 (input_text and target_text)\n",
    "#\n",
    "#\n",
    "#     100       100           100             100       100           100\n",
    "# |/(.....)\\ /(.....)\\ ... /(.....)\\| ... |/(.....)\\ /(.....)\\ ... /(.....)\\|  <-- input_text\n",
    "# |\\(.....)/ \\(.....)/     \\(.....)/| ... |\\(.....)/ \\(.....)/     \\(.....)/|  <-- target_text\n",
    "#\n",
    "# <------------- 64 ---------------->     <------------- 64 ---------------->\n",
    "#\n",
    "# <--------------------------------- 172 ----------------------------------->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ghB-VwLlD-Oz"
   },
   "source": [
    "## Build the model\n",
    "\n",
    "Use [tf.keras.Sequential](https://www.tensorflow.org/api_docs/python/tf/keras/Sequential) to define the model. For this simple example three layers are used to define our model:\n",
    "\n",
    "- [tf.keras.layers.Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding): The input layer. A trainable lookup table that will map the numbers of each character to a vector with `embedding_dim` dimensions;\n",
    "- [tf.keras.layers.LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM): A type of RNN with size units=rnn_units (You can also use a GRU layer here.)\n",
    "- [tf.keras.layers.Dense](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense): The output layer, with vocab_size outputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 425
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9280,
     "status": "ok",
     "timestamp": 1584519811503,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "0cg8DlO3QjuT",
    "outputId": "21a9daf0-1aca-4923-958e-317293023462"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tmp_input_array shape: (2, 8)\n",
      "tmp_input_array:\n",
      "[[9 0 2 1 4 6 8 8]\n",
      " [3 7 9 1 0 4 8 6]]\n",
      "\n",
      "tmp_output_array shape: (2, 8, 5)\n",
      "tmp_output_array:\n",
      "[[[-0.01348125  0.04649678  0.03382711  0.03225223  0.00077888]\n",
      "  [ 0.01548834  0.02294052 -0.03144439 -0.02458044 -0.01775809]\n",
      "  [-0.01713194  0.02779663  0.02824989 -0.01959893 -0.01335952]\n",
      "  [-0.00085108 -0.04381631  0.04934248  0.01773458 -0.0059096 ]\n",
      "  [-0.04360831  0.02180418  0.04860454 -0.04669809  0.0012654 ]\n",
      "  [-0.04296128  0.00703833 -0.01924504 -0.03515936 -0.00462629]\n",
      "  [ 0.03757649  0.01776652  0.0135636  -0.00759625  0.01923021]\n",
      "  [ 0.03757649  0.01776652  0.0135636  -0.00759625  0.01923021]]\n",
      "\n",
      " [[-0.00344205 -0.04512044 -0.04339879 -0.04655287  0.04151651]\n",
      "  [-0.01463156 -0.04065967 -0.03041167 -0.03633127 -0.04055873]\n",
      "  [-0.01348125  0.04649678  0.03382711  0.03225223  0.00077888]\n",
      "  [-0.00085108 -0.04381631  0.04934248  0.01773458 -0.0059096 ]\n",
      "  [ 0.01548834  0.02294052 -0.03144439 -0.02458044 -0.01775809]\n",
      "  [-0.04360831  0.02180418  0.04860454 -0.04669809  0.0012654 ]\n",
      "  [ 0.03757649  0.01776652  0.0135636  -0.00759625  0.01923021]\n",
      "  [-0.04296128  0.00703833 -0.01924504 -0.03515936 -0.00462629]]]\n"
     ]
    }
   ],
   "source": [
    "# Let's do a quick detour and see how Embeding layer works.\n",
    "# It takes several char indices sequences (batch) as an input.\n",
    "# It encodes every character of every sequence to a vector of tmp_embeding_size length.\n",
    "tmp_vocab_size = 10\n",
    "tmp_embeding_size = 5\n",
    "tmp_input_length = 8\n",
    "tmp_batch_size = 2\n",
    "\n",
    "tmp_model = tf.keras.models.Sequential()\n",
    "tmp_model.add(tf.keras.layers.Embedding(\n",
    "  input_dim=tmp_vocab_size,\n",
    "  output_dim=tmp_embeding_size,\n",
    "  input_length=tmp_input_length\n",
    "))\n",
    "# The model will take as input an integer matrix of size (batch, input_length).\n",
    "# The largest integer (i.e. word index) in the input should be no larger than 9 (tmp_vocab_size).\n",
    "# Now model.output_shape == (None, 10, 64), where None is the batch dimension.\n",
    "tmp_input_array = np.random.randint(\n",
    "  low=0,\n",
    "  high=tmp_vocab_size,\n",
    "  size=(tmp_batch_size, tmp_input_length)\n",
    ")\n",
    "tmp_model.compile('rmsprop', 'mse')\n",
    "tmp_output_array = tmp_model.predict(tmp_input_array)\n",
    "\n",
    "print('tmp_input_array shape:', tmp_input_array.shape)\n",
    "print('tmp_input_array:')\n",
    "print(tmp_input_array)\n",
    "print()\n",
    "print('tmp_output_array shape:', tmp_output_array.shape)\n",
    "print('tmp_output_array:')\n",
    "print(tmp_output_array)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "I7ZuvZHBD_pS"
   },
   "outputs": [],
   "source": [
    "# Length of the vocabulary in chars.\n",
    "vocab_size = len(vocab)\n",
    "\n",
    "# The embedding dimension.\n",
    "embedding_dim = 256\n",
    "\n",
    "# Number of RNN units.\n",
    "rnn_units = 1024"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "-sojdDCAICWO"
   },
   "outputs": [],
   "source": [
    "def build_model(vocab_size, embedding_dim, rnn_units, batch_size):\n",
    "    model = tf.keras.models.Sequential()\n",
    "\n",
    "    model.add(tf.keras.layers.Embedding(\n",
    "      input_dim=vocab_size,\n",
    "      output_dim=embedding_dim,\n",
    "      batch_input_shape=[batch_size, None]\n",
    "    ))\n",
    "\n",
    "    model.add(tf.keras.layers.LSTM(\n",
    "      units=rnn_units,\n",
    "      return_sequences=True,\n",
    "      stateful=True,\n",
    "      recurrent_initializer=tf.keras.initializers.GlorotNormal()\n",
    "    ))\n",
    "\n",
    "    model.add(tf.keras.layers.Dense(vocab_size))\n",
    "  \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "XoPwxyAPEg6z"
   },
   "outputs": [],
   "source": [
    "model = build_model(vocab_size, embedding_dim, rnn_units, BATCH_SIZE)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8897,
     "status": "ok",
     "timestamp": 1584519812083,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "iLnlZFgU55bQ",
    "outputId": "eeece968-3a2a-4652-a0b7-d8f6944cdbc2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (64, None, 256)           16640     \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (64, None, 1024)          5246976   \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (64, None, 65)            66625     \n",
      "=================================================================\n",
      "Total params: 5,330,241\n",
      "Trainable params: 5,330,241\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 422
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8704,
     "status": "ok",
     "timestamp": 1584519812084,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "CcaO_rO_8-GH",
    "outputId": "21830f00-dc89-417a-81d8-8988354235f4"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAb0AAAGVCAIAAADls7hIAAAABmJLR0QA/wD/AP+gvaeTAAAgAElE\nQVR4nOzdeVhT17ow8LUhhBCGADJIwSCDIJPidDUIeihHa0UZnEBrW3qetor6BdSjFNAKKGirRS4W\n7FPL4bRVwQEvqMix14EKVtSKCMXWAoqonDKIDJJAEpLvj3W7by5DyE5CEuD9/eWe1lp7N7zdw1rv\nIiQSCQIAACA3HU03AAAARhmImwAAQA3ETQAAoAbiJgAAUEOTXrh161ZqaqqmmgIAANqJw+Fs27aN\nXPw/95vPnj07e/as2psEwCh29uzZ58+fa7oVI66srKysrEzTrdCMsrKyW7duSa+hDdzpzJkz6moP\nAKMeQRBbt25ds2aNphsyslavXo3Ga3DA5y4N3m8CAAA1EDcBAIAaiJsAAEANxE0AAKAG4iYAAFAD\ncRMADbh06RKLxbpw4YKmG6JiGzduJP60fv166U1XrlyJjY3F/xYKhSkpKc7OznQ63dTU1NPTs76+\nfmBpPT09U6dO3bVrlzxVi8Xi0NBQNpvNYDBsbW2Dg4MrKysRQufPn//ss8/6+vrIPfPz88lGWlhY\nKHCaEDcB0IAxnIfM3Ny8qKjo0aNHWVlZ5Mo9e/akp6fHxcXhxbCwsO++++7EiRM8Hu/XX391cnJ6\n/fr1wKLi4+MfPXokZ71isbikpOTkyZNtbW2lpaV8Pn/BggWNjY1BQUEMBiMgIKC9vR3vGRwc/Pz5\n8xs3bixdulSxc4S4CYAGBAYGdnR0LF++fKQr4vP5Pj4+I12LNAMDgyVLlri4uOjr6+M1Bw4cyM3N\nPX36tLGxMUIoNzc3Pz//zJkzc+fOpdFoNjY2BQUFnp6e/cr56aeffvnlF0pVczgcX19fJpPp4OCQ\nnJzc0dHxz3/+EyEUFRU1ffr0pUuXikQihBBBELa2tn5+flOmTFHsHCFuAjCWZWVlNTc3a7ABtbW1\nu3fvTkxMZDAYeM3Ro0dnzpzp5eUl4yg+n79jx460tDT5K6LRaNLvPRwdHRFCdXV1eDEhIaGiooJS\ngTJA3ARA3UpLS9lsNkEQX375JUIoMzPT0NCQyWQWFBS8/fbbJiYmdnZ2OTk5eOf09HQGg2FlZbVx\n40YbGxsGg+Hj43P79m28lcvl0un0iRMn4sXNmzcbGhoSBNHa2ooQio6O3r59e11dHUEQzs7OCKF/\n/etfJiYmycnJajvZ9PR0iUQSFBSEFwUCQVlZmbe3t+yj4uPjN2/ebGlpqXC9fD4fIWRiYoIXzczM\nFi5cmJaWppI3JBA3AVA3X1/fn376iVzctGnT1q1b+Xy+sbHxqVOn6urqHB0dP/roI6FQiBDicrkR\nERE8Hi8qKqq+vr68vFwkEi1atOjZs2cIofT0dOkhnhkZGYmJieRiWlra8uXLnZycJBJJbW0tQgh/\nHhGLxWo72cLCQldXVyaTiRcbGxsFAsG9e/f8/f3x/wbc3NwyMjKkw9nNmzfr6urWrVunTL137txB\nCPn6+pJrZsyY8eLFiwcPHihTLAZxEwBt4ePjY2JiYmlpGR4e3t3d3dDQQG6i0Whubm76+vru7u6Z\nmZldXV3Z2dkKVBEYGNjZ2bl7927VtVqW7u7uJ0+eODk5kWvw9x9LS8vk5OTq6uqmpqaQkJAtW7ac\nPHkS78Dn86OjozMzMxWutKmpKTc3NyoqisPhkPe5CCH8NrOqqkrhkkkQNwHQOnQ6HSGE7zcHmj17\nNpPJ/O2339TbKEU0NzdLJBLyZhMhhD8WeXh4+Pj4mJubs1isxMREFov19ddf4x3i4uI+/vhjW1tb\nhSvlcDhRUVEhISFFRUV6enrketyMpqYmhUsmDZIPCQCg5fT19VtaWjTdiuH19PSgP2MlZmNjgxDC\nr18xOp1ub2+PP+CUlpZWVVUpmQXYysoqKyvLw8Oj33oDAwOySUqC+00ARhmhUNje3m5nZ6fphgwP\nhyrpPudGRkZTpkx5+PCh9G4ikYjFYiGEsrKyrl69qqOjgzul4+9CycnJBEH8/PPPclZqaWlpamo6\ncL1AICCbpCSImwCMMsXFxRKJZN68eXiRRqMN9USvcVZWVgRBdHR0SK8MCwu7f//+48eP8SKPx3v6\n9CnulpSdnS2Rgu+p4+PjJRLJ7Nmz5az0woULgz7m42ZYW1src0YYxE0ARgGxWPzq1SuRSFRZWRkd\nHc1msyMiIvAmZ2fntra2/Px8oVDY0tLy9OlT6QPNzc0bGxvr6+u7urqEQmFRUZE6+yExmUxHR8d+\n+fC3bdtmb28fERHR0NDw8uXLmJgYPp//ySefyFNgeHi4tbV1eXn5UDvU1tZaW1uHhYUN3ISbIbvf\nqJwgbgKgbl9++eWcOXMQQjExMcHBwZmZmYcPH0YITZs27fHjx8eOHdu+fTtCaMmSJTU1NfiQnp4e\nLy8vAwMDPz8/FxeX69evky8NN23a5O/vv3btWldX17179+LnUA6HgzsqRUZGWllZubu7L126tK2t\nTf0nGxgYWF1djXtTYmZmZiUlJXZ2dt7e3ra2tnfu3CksLBy2RycmEAiam5sLCgqG2kFG98y7d+/a\n2tpOmzaNUvuHrIZ06tSpfmsAALIhhE6dOjWiVWzYsMHc3HxEqxjWqlWrVq1aNexuGzZssLW1lV5T\nU1NDo9G+//57lTSjr6/Pz88vKyuL6oGtra0MBuPQoUPSK6OioiZMmDDssQPPHe43ARgFpD+taDk+\nn3/58uWamhr8HcbZ2TkpKSkpKWnQzB2U9PX15efnd3V1hYeHUz02ISHB29uby+UihCQSSWNjY2lp\nKR4LoACImwAAVWpra8N5Pf72t7/hNbGxsatXrw4PD+/3gYiq4uLivLy8oqIi6Q6h8khNTa2oqLh0\n6RLuzllQUIDzehQWFirWEs3EzTlz5ujq6sr5RmMoH374obGxMUEQFRUV8mxVc8ZDsVh8+PBhqqlo\ntDAtY1lZmZubG+4aYm1tvW/fPrVVnZeX5+joiLukTJw4sV8+x3EiLi4uOzu7o6PDwcFB+6fp/uqr\nr8iH2ePHj5Prk5OTuVzu/v37lSk8ICDgxIkT5GB8ORUUFPT29hYXF5uZmeE1ISEh0s/vijRF+qFd\nne83AwICpk+frmQhOPfB/fv35dl68eJFExOT8+fPK1mpPH7//ff58+cjhKieozobSclbb72FEHr1\n6pX6q3ZycmKxWOqvV05o5N9vagM532+OSQPPXZPjhQiCUGd1OOOhGip68OBBUlJSZGRkd3e3hGLy\nFbU1ks/nBwQESGeX0BJa2zAASJp8vyk9dFQxsiOvCuOyRCI5c+YMOYRWtunTp+fl5b3zzjvSw8u0\njcbTMg5FaxsGAEmRuNnX1/fpp5+y2WwDA4Np06bhp/u0tDRDQ0MdHZ1Zs2ZZW1vr6ekZGhrOnDnT\nz89v0qRJDAbD1NR0586d0uXU1tZOnTrV0NAQ90orLS2VXQVCSCKRHDx40NXVVV9fn8Vi7dixQ7pA\nGVspZTzEDUhJSXF1dTUwMLCwsHBwcEhJSZFO2DUSRktaRnU2TB4lJSXu7u4sFovBYHh5eV2+fBkh\n9OGHH+IXo05OTvfv30cIffDBB0wmk8VinT9/Hg3xG/v888+ZTKaxsXFzc/P27dttbW3ln6cBjCPS\nD+1yvt/8+9//rq+vf/bs2VevXsXFxeno6Ny9e1cikezZswchdPv27e7u7tbW1iVLliCECgsLW1pa\nuru7cQ+AiooKXEhAQICjo+OTJ0+EQuEvv/wyd+5cBoPx+++/y64iPj6eIIgvvvji1atXPB4vIyMD\nSb3BlL0VdwM+cuQIuTNC6OrVqx0dHc3NzX5+foaGhgKBAG9NTk7W1dUtKCjg8Xj37t2ztrb+y1/+\nQvW1yNy5c6m+36TUyA0bNhgaGj58+LCnp6e6unrOnDnGxsYNDQ146zvvvGNtbU2WfPDgQYRQS0sL\nXly5ciVOy4hdvHjR2Ng4KSlpqIb1e7+ptoZJ5Hi/eebMmYSEhLa2tpcvX86bN4/skbdy5UpdXd0X\nL16Qe65bt458dyzjN4YQioqKOnLkyIoVK3799VcZVUvg/eY4oIL+mz09PZmZmaGhoStXrjQ1Nd21\na5eenp50KkB3d3cmkzlhwoS1a9cihNhstoWFBZPJxB9DpZNfGRsbT548mUajeXh4HDt2rKenBz8I\nD1UFn88/fPjwX//6123btpmamhoYGJibm5Olyd46lKEyHubn58+aNSsoKMjAwGDmzJnBwcE3btzA\n/dHUT2vTMqqhYfJYtWrVnj17zMzMzM3Ng4KCXr58icc1R0ZG9vX1kfV2dnbevXsXT8U17M/4wIED\nW7ZsycvLmzp16gg1G4xelL8LPXr0iMfjkZMoGRgYTJw4cdBUgDiHIJ4ICf35NnOoBAReXl4sFgvP\n2zlUFbW1tTweLyAgYNASZG8dVr+Mhz09PeR0KAihvr4+PT09XV1dxQpXFa1Ny6g9DcM/M9xL/M03\n33RxcfnHP/4RFxdHEERubm54eDj+jyj/z1geYWFhgw6IHnvU/C1Xe6xatUp6kXLc7O7uRgjt2rVL\nelJjnFNPSXp6evgPb6gq8LD8oaYckb2VqqVLlx48eLCgoGDx4sXV1dX5+fnLli3TeNwcltamZRzR\nhhUWFh48eLC6urqzs1M6dhMEsXHjxm3btl29evWvf/0rnngWb1Ltzzg6OprD4ShxBqMAHkG/detW\nTTdEA/C5S6McN3FgOnz4cHR0tGoahRBCSCQStbW1sdlsGVVcv34dIdTb2ztoCfj2cKitVCUkJNy7\ndy8iIuL169c2NjZr1qxR51RWitHatIwj0bAbN27cu3dv69atDQ0NoaGhK1as+Mc//vHGG28cOXJE\n+vNjREREXFzcN998M2nSJBMTE3t7e7xetT9jDocz0t8MNe7MmTMIoTF/moPC5y6NctzEH8cHHaKj\njOvXr4vF4pkzZ8qowtPTU0dH58cff4yMjBxYguytVFVXV9fV1bW0tNBooyYlvtamZRyJht27d8/Q\n0BAhVFVVJRQKN23ahOd97fcgaWZmFhYWlpuba2xs/NFHH5HrR+hnDMYJyt+FGAzGBx98kJOTk5mZ\n2dnZ2dfX9/z583//+98K1C0QCDo6OkQiUXl5OZfLxSn5ZFRhaWm5cuXKs2fPZmVldXZ2VlZWSneo\nlL2Vqi1btrDZbOUzEYw0rU3LqKqGDSxZKBQ2NTUVFxfjuImfUa5cudLT01NTU0N2eCJFRkb29vZe\nvHhx+fLl5EoV/ozBeCT9cV3Ofki9vb0xMTFsNptGo+FoVV1dnZaWhgfbT548uaSk5MCBAzjxvbW1\n9YkTJ3Jzc3GaZTMzs5ycHIlEkp2d7e/vb2VlRaPR8Mf3p0+fyq5CIpF0dXV9+OGHEyZMMDIy8vX1\n/fTTTxFCdnZ2Dx48kL31yJEjuM8gk8kMCgrKyMjArZ0yZUpdXd3XX3+N51m2t7fHfaGuXbs2YcIE\n8irp6em5ubnl5eXJ02vh1q1b8+fPJ1+WTZw40cfH58cffxz2QKqN3LBhg56enq2tLY1GMzExCQkJ\nqaurI0t7+fKlv78/g8FwcHD4f//v/+HerM7Ozrg/UHl5ub29vYGBga+v7x9//HHp0iVjY+N9+/YN\nbFVZWZmHh4eOjg4+l+TkZLU17OjRo9JTIfZz7tw5XGBMTIy5ubmpqenq1atx11cnJyey25NEIpkx\nY0ZsbGy/8xr0N/bZZ5/h/JWTJk2SM/UZgn5IY93Ac4f8m4PLyMiIjo4mF3t7e7du3aqvr8/j8TTY\nqn60IS3joLStYUuXLn38+PEIFQ5xc8zTrvHpWuuPP/7gcrnSL7/odDqbzRYKhUKhUCXzOqmK1qZl\n1HjDhEIh7pNUWVmJ72012x4wlkD+zUEYGBjo6ellZWU1NTUJhcLGxsZvvvnm008/DQ8Pb2xsJIYm\nO53qb7/9pvCxgKqYmJiamprff//9gw8+2Lt3r6abM15s3LiR/En3y/t35cqV2NhY/G+hUJiSkuLs\n7Eyn001NTT09Pevr6weW1tPTM3XqVOm+YjKIxeLQ0FA2m81gMGxtbYODg3F/8PPnz3/22WfS/yPP\nz88nG2lhYaHIeUrffMJzOunGjRt//etfTUxMdHV1WSyWj49PRkaGUCjUdLv+V2xsLO5tPnny5DNn\nzmi6Of9LSxoWHx+vo6MzadKkkU7Kh+A5XQp+RVNUVPTo0aOenh5y/aeffrp8+fLOzk68GBoa6urq\nWlZWhu9LgoKCqqqqBpa2bds29Od8lsMSCoUTJkwoKSnp7u5+/PjxokWLWCwWHmWblpa2cOFCcpSw\nWCx+/vz5jRs3li5dqtg8GRA3AVDKSMdNHo/H4XA0XpTC8wtJJJL9+/e7uLjw+Xy8mJOTQxBEZWWl\n7KJu3ry5ePFiSnFz2bJl5OKdO3cQQsnJyXiRy+VyOJx+tz4wvxAAY5MKE+tpJEdfbW3t7t27ExMT\nyYHLR48enTlzpuz5ePl8/o4dO9LS0uSviEajSU+UgPvz1tXV4cWEhISKigpKBcoAcROAESeRSFJT\nU3GiEzMzs5CQEHIsPKXEehpMHqiw9PR0iUQSFBSEFwUCQVlZ2bBz5MTHx2/evFmZYdN45mHcQw4h\nZGZmtnDhwrS0NAnFVOKDgrgJwIhLSEiIjY2Nj49vbm6+cePGs2fP/Pz8mpqaEELp6enSgxczMjIS\nExPJxbS0tOXLl+PEerW1tVwuNyIigsfjRUVF1dfXl5eXi0SiRYsW4fSDlIpCf/Z5EIvFI3ruhYWF\nrq6u5ExqjY2NAoHg3r17/v7+OPS7ubllZGRIh7ObN2/W1dWtW7dOmXrxc7qvry+5ZsaMGS9evHjw\n4IEyxWIQNwEYWXw+PzU1dcWKFevXr2exWF5eXl999VVra6vC49k0mzyQku7u7idPnkgPXsBj8Cwt\nLZOTk6urq5uamkJCQrZs2XLy5Em8A5/Pj46OzszMVLjSpqam3NzcqKgoDodD3ucihKZMmYIQqqqq\nUrhkEsRNAEZWdXX169evZ8+eTa6ZM2cOnU4fOCRUARpMHiiP5uZmiUQiPW0vnjzGw8PDx8fH3Nyc\nxWIlJiayWCzy/yJxcXEff/yxra2twpVyOJyoqKiQkJCioiLpyXhwM/BtvpKg3zsAI6u9vR0hZGRk\nJL3S1NS0q6tLJeVrbfJAhFBPTw/6M1ZiePyx9Oy7dDrd3t4ef8ApLS2tqqpKTU1VplIrK6usrCwP\nD49+6/GIFdwkJcH9JgAjy9TUFCHUL0qqKrGe1iYPxHCoku5zbmRkNGXKlIcPH0rvJhKJcDqLrKys\nq1ev6ujo4E7p+LtQcnIyQRA///yznJVaWlria94Pnq9BJeP9IG4CMLI8PT2NjIyk/+xv374tEAhm\nzZqFF5VJrKe1yQMxKysrgiD6TW0dFhZ2//79x48f40Uej/f06VPcLSk7O1u6myS+j8b9N6VfdMh2\n4cKFQR/zcTNwgiElQdwEYGQxGIzt27efO3fu+PHjnZ2dVVVVkZGRNjY2GzZswDtQTayntckDB2Iy\nmY6OjnguBtK2bdtw0siGhoaXL1/GxMTw+fxPPvlEngLDw8Otra3Ly8uH2qG2ttba2nrQaUtwM2T3\nG5UTxE0ARtyePXtSUlKSkpIsLCwWLlw4efJkMn8oQmjTpk3+/v5r1651dXXdu3cvfpDkcDi4d1Fk\nZKSVlZW7u/vSpUvb2toQQj09PV5eXnj2bBcXl+vXr5MvEKkWpQaBgYHV1dW4NyVmZmZWUlJiZ2fn\n7e1ta2t7586dwsLCYXt0YgKBoLm5uaCgYKgdZHTPvHv3rq2t7bRp0yi1f8hqSDDOEgCqkHrHp2sq\nR5/C4yxrampoNJqcyUyH1dfX5+fnl5WVRfXA1tZWBoNx6NAh6ZUwzhKA8ULjOfpk4/P5ly9frqmp\nwd9hnJ2dk5KSkpKSlJ89oa+vLz8/v6urS4HkYQkJCd7e3lwuFyEkkUgaGxtLS0tx/38FQNwEAKhS\nW1vbkiVLXFxc/va3v+E1sbGxq1evDg8P7/eBiKri4uK8vLyioiLpDqHySE1NraiouHTpEu7OWVBQ\nYGtr6+fnV1hYqFhLIG4CMGrExcVlZ2d3dHQ4ODicPXtW080ZxFdffUU+zB4/fpxcn5yczOVy9+/f\nr0zhAQEBJ06cIAfgy6mgoKC3t7e4uNjMzAyvCQkJkX5+V6Al0O8dgFEjJSUlJSVF061Q0OLFi3Fe\nODULDg4ODg5WbZlwvwkAANRA3AQAAGogbgIAADUQNwEAgJpBvgudPn1a/e0AYPS6deuWppsw4vAg\nxfEZHJ4/f94/c4p0J3g8XggAAIC0fuOFCIkqZtsAQOUIgjh16pT0xA8AaAl4vwkAANRA3AQAAGog\nbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANxEwAAqIG4CQAA1EDcBAAAaiBuAgAANRA3AQCA\nGoibAABADcRNAACgBuImAABQA3ETAACogbgJAADUQNwEAABqIG4CAAA1EDcBAIAaiJsAAEANxE0A\nAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGogbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANx\nEwAAqIG4CQAA1BASiUTTbQAAIYQ2bNjw6NEjcrG8vNzBwcHMzAwv6urqfvvtt3Z2dhpqHQD/i6bp\nBgDwP6ytrb/++mvpNZWVleS/HR0dIWgCLQHP6UBbrFu3bqhNdDo9IiJCjW0BQBZ4TgdaxNPT8+HD\nh4P+Jh89euTi4qL+JgEwENxvAi3y3nvv6erq9ltJEMT06dMhaALtAXETaJG1a9f29fX1W6mrq/v+\n++9rpD0ADAqe04F28fHxuX37tlgsJtcQBPHs2TNbW1sNtgoAaXC/CbTLu+++SxAEuaijo+Pr6wtB\nE2gViJtAu6xevVp6kSCI9957T1ONAWBQEDeBdrGwsAgICCC/DhEEERoaqtkmAdAPxE2gddavX49f\nu+vq6r711lsTJkzQdIsA+D8gbgKts2LFCjqdjhCSSCTr16/XdHMA6A/iJtA6hoaGy5YtQwjR6fTl\ny5drujkA9AdxE2ijd955ByEUGhpqaGio6bYAMIBEPqtWrdJ0SwEAYGTJGQ8p5EOaN2/e1q1bR67F\nAEg7fvx4eHg4jUY5Zdfhw4cRQmP+t3rr1q20tLRTp05puiFjBL6ecu5M4UdpZ2e3Zs0ahZoEAGVB\nQUEMBkOBA8+cOYMQGg+/1bS0tPFwmmojf9yE95tASykWNAFQA4ibAABADcRNAACgBuImAABQA3ET\nAACogbgJAEIIXbp0icViXbhwQdMN0UZXrlyJjY3F/xYKhSkpKc7OznQ63dTU1NPTs76+fuAhPT09\nU6dO3bVrlzzli8Xi0NBQNpvNYDBsbW2Dg4PxlHznz5//7LPPBqay1jiImwAghBAk8B7Knj170tPT\n4+Li8GJYWNh333134sQJHo/366+/Ojk5vX79euBR8fHx0rM6yyYWi0tKSk6ePNnW1lZaWsrn8xcs\nWNDY2Ij7ogUEBLS3t6vsfFQB4iYACCEUGBjY0dGhhuHwfD7fx8dnpGtRlQMHDuTm5p4+fdrY2Bgh\nlJubm5+ff+bMmblz59JoNBsbm4KCAk9Pz35H/fTTT7/88gulijgcjq+vL5PJdHBwSE5O7ujo+Oc/\n/4kQioqKmj59+tKlS0UikYrOSQUgbgKgVllZWc3NzZpuhVxqa2t3796dmJhI9qU9evTozJkzvby8\nZBzF5/N37Nghfx9yhBCNRpN+Q+Lo6IgQqqurw4sJCQkVFRWUChxpEDcBQKWlpWw2myCIL7/8EiGU\nmZlpaGjIZDILCgrefvttExMTOzu7nJwcvHN6ejqDwbCystq4caONjQ2DwcBzIuGtXC6XTqdPnDgR\nL27evNnQ0JAgiNbWVoRQdHT09u3b6+rqCIJwdnZGCP3rX/8yMTFJTk7WwGkPJz09XSKRBAUF4UWB\nQFBWVubt7S37qPj4+M2bN1taWipcL5/PRwiZmJjgRTMzs4ULF6alpWnPuxSImwAgX1/fn376iVzc\ntGnT1q1b+Xy+sbHxqVOn6urqHB0dP/roI6FQiBDicrkRERE8Hi8qKqq+vr68vFwkEi1atOjZs2cI\nofT0dOmxjxkZGYmJieRiWlra8uXLnZycJBJJbW0tQgh/9JCeh057FBYWurq6MplMvNjY2CgQCO7d\nu+fv74//h+Hm5paRkSEdzm7evFlXV7du3Tpl6r1z5w5CyNfXl1wzY8aMFy9ePHjwQJliVQjiJgBD\n8vHxMTExsbS0DA8P7+7ubmhoIDfRaDQ3Nzd9fX13d/fMzMyurq7s7GwFqggMDOzs7Ny9e7fqWq0a\n3d3dT548cXJyItfg7z+WlpbJycnV1dVNTU0hISFbtmw5efIk3oHP50dHR2dmZipcaVNTU25ublRU\nFIfDIe9zEUJTpkxBCFVVVSlcsmpB3ARgeDj/PL7fHGj27NlMJvO3335Tb6NGVnNzs0QiIW82EUL6\n+voIIQ8PDx8fH3NzcxaLlZiYyGKxvv76a7xDXFzcxx9/rMzkoxwOJyoqKiQkpKioSE9Pj1yPm9HU\n1KRwyapFOUkXAGAgfX39lpYWTbdClXp6etCfsRKzsbFBCOEXtRidTre3t8cfcEpLS6uqqlJTU5Wp\n1MrKKisry8PDo996AwMDsknaAO43AVCWUChsb2+3s7PTdENUCYcq6T7nRkZGU6ZMefjwofRuIpGI\nxWIhhLKysq5evaqjo0MQBEEQ+LtQcnIyQRA///yznJVaWlqampoOXC8QCMgmaQOImwAoq7i4WCKR\nzJs3Dy/SaLShnuhHESsrK4IgOjo6pFeGhYXdv3//8ePHeJHH4z19+hR3S8rOzpbOiI7vvuPj4yUS\nyezZs+Ws9MKFC4M+5uNmWFtbK3NGKgRxEwBFiMXiV69eiUSiysrK6OhoNpsdERGBNzk7O7e1teXn\n5wuFwpaWlqdPn0ofaG5u3tjYWF9f39XVJRQKi4qKtLMfEpPJdHR0fP78ucULfuUAACAASURBVPTK\nbdu22dvbR0RENDQ0vHz5MiYmhs/nf/LJJ/IUGB4ebm1tXV5ePtQOtbW11tbWYWFhAzfhZsjuN6pO\nEDcBQF9++eWcOXMQQjExMcHBwZmZmXiyjWnTpj1+/PjYsWPbt29HCC1ZsqSmpgYf0tPT4+XlZWBg\n4Ofn5+Licv36dfJV4KZNm/z9/deuXevq6rp37178dMnhcHBHpcjISCsrK3d396VLl7a1tWnkfOUU\nGBhYXV2Ne1NiZmZmJSUldnZ23t7etra2d+7cKSwsHLZHJyYQCJqbmwsKCobaQUb3zLt379ra2k6b\nNo1S+0eQ/POyrVq1Ss6dAdAgNfxWN2zYYG5uPqJVDAvPLDSiVdTU1NBotO+//14lpfX19fn5+WVl\nZVE9sLW1lcFgHDp0SCXNGAql6wn3mwAoQguT9Kics7NzUlJSUlLSoJk7KOnr68vPz+/q6goPD6d6\nbEJCgre3N5fLVbINKqR1cXPOnDm6urpy3vkP5cMPPzQ2NiYIoqKiQp6tas4hJhaLDx8+TCm5Q15e\nnqOjIzGYyZMnK9CG8XCdgfJiY2NXr14dHh7e7wMRVcXFxXl5eUVFRdIdQuWRmppaUVFx6dIl6e6c\nGqd1cfPu3bv+/v5KFvLNN98cO3ZM/q0SNY57rampWbBgwbZt23g8nvxHrVy58vHjx05OTiwWCz8p\niEQiHo/X1NRE9YeIjfnrPHLi4uKys7M7OjocHBzOnj2r6eaMuOTkZC6Xu3//fmUKCQgIOHHiBDls\nX04FBQW9vb3FxcVmZmbK1K5yWtrvnSAIdVaHc4ipoaIHDx4kJSVFRkZ2d3crGUR0dXUNDAwMDAxc\nXFwULmSsXucRlZKSkpKSoulWqNXixYsXL16s/nqDg4ODg4PVX++wtO5+E1P+nlx2RFBhvJBIJGfO\nnCGHmsk2ffr0vLy8d955R3oYhpLy8/MVPnasXmcARpSK42ZfX9+nn37KZrMNDAymTZuGP1GlpaUZ\nGhrq6OjMmjXL2tpaT0/P0NBw5syZfn5+kyZNYjAYpqamO3fulC6ntrZ26tSphoaGuJ9HaWmp7CoQ\nQhKJ5ODBg66urvr6+iwWa8eOHdIFythKKYcYbkBKSoqrq6uBgYGFhYWDg0NKSop0ChyFKZlSDK4z\nAGoi53d3Oft2/P3vf9fX1z979uyrV6/i4uJ0dHTu3r0rkUj27NmDELp9+3Z3d3dra+uSJUsQQoWF\nhS0tLd3d3fhLWUVFBS4kICDA0dHxyZMnQqHwl19+mTt3LoPB+P3332VXER8fTxDEF1988erVKx6P\nl5GRgRC6f/8+Pkr2Vtyx7siRI+TOCKGrV692dHQ0Nzf7+fkZGhoKBAK8NTk5WVdXt6CggMfj3bt3\nz9ra+i9/+Yucl5E0d+7c6dOn91t58eJFY2PjpKSkoY6Sfr8pkUiioqKqqqqkd4DrLBk3febU0A9p\nXKF0PVUZN/l8PpPJDA8Px4s8Hk9fX3/Tpk2SP/+eu7q68KZvv/0WIUT+zeN0e7m5uXgxICBAOqbg\nGZr+/ve/y6iCx+MxmcxFixaRR+E7F/wXK3urZIi/Zz6fjxfxH39tbS1enDNnzn/8x3+QRX388cc6\nOjq9vb1yXMX/NWjcHJZ0Ui9s0Lg5zq8zxE2gAErXU5XfhR49esTj8cjJRgwMDCZOnDhoci2clYuc\nMAS/ZRtqSK+XlxeLxcJ/1UNVUVtby+PxAgICBi1B9tZh9csh1tPTQ04bgBDq6+vT09PT1dVVrHCq\nWCwWOUdVdHS07J3H7XV+/vz56dOnFWvGaHHr1i2E0Jg/TbXB11NOqoyb3d3dCKFdu3ZJT/6Jc08p\nSU9PD/85DVUFHr46VGp+2VupWrp06cGDBwsKChYvXlxdXZ2fn79s2TK1xU1pKp9xZcxc57KyskGH\nOY894+Q0tY0qvwvhP5jDhw9L39BSiuKDEolEbW1tbDZbRhX4xqS3t3fQEmRvpSohIeHNN9+MiIgw\nMTFZsWLFmjVrZPRhHEXG0nWG53RAFfnlUx6qjJv4o+2gQ0eUcf36dbFYPHPmTBlVeHp66ujo/Pjj\nj4OWIHsrVdXV1XV1dS0tLUKhsKGhITMzU7Odcv/9739/8MEHypcD1xkAOakybjIYjA8++CAnJycz\nM7Ozs7Ovr+/58+f//ve/FShKIBB0dHSIRKLy8nIul4tTV8mowtLScuXKlWfPns3Kyurs7KysrJTu\n6Cd7K1Vbtmxhs9nKj9gdiGpKMYlEwufz8/LyyJn/qBqf1xkAZcl5EyvnN8re3t6YmBg2m02j0fBf\nUXV1dVpaGh4LOHny5JKSkgMHDuAE0dbW1idOnMjNzcXpSM3MzHJyciQSSXZ2tr+/v5WVFY1GmzBh\nwtq1a58+fSq7ColE0tXV9eGHH06YMMHIyMjX1/fTTz9FCNnZ2T148ED21iNHjuDhX0wmMygoKCMj\nA7d2ypQpdXV1X3/9NY5K9vb2uI/OtWvXJkyYQF5APT09Nze3vLw8eS7jrVu35s+fT77znThxoo+P\nz48//oi3Xrp0ydjYeN++fQMPPHfu3MCP6aRdu3ZJJBK4zhh8TwcK0Fg/pHEiIyMjOjqaXOzt7d26\ndau+vj6Px9Ngq8Yeha/zOPmtQtxULY31QxoP/vjjDy6XK/3ij06ns9lsoVAoFAq1Z/6T0Q6uM9Bm\nWjo+XWsZGBjo6ellZWU1NTUJhcLGxsZvvvnm008/DQ8Pb2xsHDTPG6ZA2sHxTMZ1VvhlLgCqAnGT\nGhaL9cMPP/zyyy8uLi4GBgbu7u7Z2dkHDhz49ttvp06dKuPGPjc3V9NtH01kXGdNN220unLlSmxs\nLP63UChMSUlxdnam0+mmpqaenp719fUDD+np6Zk6dap0H14ZxGJxaGgom81mMBi2trbBwcF4DMWw\nkpKS3N3dTUxM9PX1nZ2dd+7cKf0xcN++ff1uQcjhGLLP5fz585999tkIpZeG53TK/Pz8/vu//1vT\nrRj74Dqr0J49e+7fv3/ixAm8GBYW9vDhwxMnTsyaNaulpWXjxo2D9luIj49/9OiRnFWIxeKSkpL8\n/PyZM2c2NTVt2LBhwYIFDx8+fOONN2QfeO3atS1btoSHh+vp6RUVFa1fv76qqqqoqEjOeoc6l6Cg\noCdPngQEBOTn5w86t7BS5HwPOk7etYMxQA2/VR6Px+FwNFuU/N8x9u/f7+LiQuYByMnJIQiisrJS\n9lE3b97EOTfxXL7DEgqFy5YtIxdxMoTk5ORhDwwMDBSJROQiTnnV0NCAF/fu3StjgqNhz4XL5XI4\nHKFQOGwzYH4hAEZWVlZWc3OzthU1qNra2t27dycmJpKD/Y8ePTpz5kzZc+ry+fwdO3ZQGshLo9Gk\nZ0BxdHRECNXV1Q174MWLF6WHz1pYWCCE5JwNYdhzSUhIqKioUPmIZIibYJySSCSpqalubm76+vpm\nZmYhISFkDhoul0un08lJHTZv3mxoaEgQRGtrK0IoOjp6+/btdXV1BEE4Ozunp6czGAwrK6uNGzfa\n2NgwGAwfH5/bt28rUBRSOgfrQOnp6RKJJCgoCC8KBIKysrJh55WKj4/fvHmzMqkG8OzBCnzEe/Hi\nhYGBgYODw7B7ynMuZmZmCxcuTEtLk6h0jhaIm2CcSkhIiI2NjY+Pb25uvnHjxrNnz/z8/JqamhBC\n6enp0gmSMzIyEhMTycW0tLTly5c7OTlJJJLa2loulxsREcHj8aKiourr68vLy0Ui0aJFi3DSPEpF\noT+nyRSLxao6zcLCQldXV3ISqsbGRoFAcO/ePX9/fxzl3dzcMjIypMPKzZs36+rq1q1bp0y9+Dnd\n19eX0lE8Hu/atWsfffQRTo6FxcbGmpmZ0el0BweHkJCQu3fvyn8uCKEZM2a8ePHiwYMHypxOPxA3\nwXjE5/NTU1NXrFixfv16Fovl5eX11Vdftba2KjwwlEaj4VtXd3f3zMzMrq6u7OxsBcoJDAzs7Ozc\nvXu3Ys3op7u7+8mTJ9IjzfA3E0tLy+Tk5Orq6qamppCQkC1btpw8eRLvwOfzo6OjMzMzFa60qakp\nNzc3KiqKw+GQ97lySklJsbGx2bdvH7nm/fffP3/+/LNnz16/fp2Tk9PQ0LBw4cLq6mp5zgWbMmUK\nQqiqqkrhMxoI4iYYj6qrq1+/fj179mxyzZw5c+h0Ovl8rYzZs2czmcxBM8+qWXNzs0QikZ7xFE9s\n5eHh4ePjY25uzmKxEhMTWSwW+T+MuLi4jz/+2NbWVuFKORxOVFRUSEhIUVERpQmszp07d/r06cuX\nLxsbG5MrJ02aNGPGDCMjIzqdPm/evOzsbD6fj1NcD3suGD59/CShKtAPCYxHOPezkZGR9EpTU9Ou\nri6VlK+vr9/S0qKSopTR09OD/owvGM6NgN+uYnQ63d7eHn/AKS0traqqSk1NVaZSKyurrKwsDw8P\nSkfl5uampqYWFxfL7rfk5eWlq6v7+++/o+HOhYRHl+FLoSpwvwnGI9yhr1+UbG9vt7OzU75woVCo\nqqKUhEOGdN9vIyOjKVOmPHz4UHo3kUiEU8BkZWVdvXpVR0cH9zDH34WSk5MJgvj555/lrNTS0pJq\nf8kjR44cP3782rVrw3b2FIvFYrEY/59A9rmQBAIB+vNSqArETTAeeXp6GhkZSceC27dvCwSCWbNm\n4UUajTbUhCLDKi4ulkgk8+bNU74oJVlZWREE0W/O+rCwsPv37z9+/Bgv8ni8p0+f4q482dnZ0r0U\n8S0z7r8p/U5DtgsXLsj/mC+RSGJiYqqqqvLz8/vd/mNvvfWW9CKeHJDD4Qx7LiR8+jgZmKpA3ATj\nEYPB2L59+7lz544fP97Z2VlVVRUZGWljY7Nhwwa8g7Ozc1tbW35+vlAobGlpefr0qfTh5ubmjY2N\n9fX1XV1dOCaKxeJXr16JRKLKysro6Gg2m40zmVItimoOVtmYTKajoyOev4S0bds2nGi1oaHh5cuX\nMTExfD7/k08+kafA8PBwa2vr8vLyoXaora21trbuN3uHjKMePnz4+eefHzt2TE9PT3ow5aFDh/AO\nL168yM3NbW9vFwqFt27d+vDDD9lsdmRkpPzngk9fdn9VqiBugnFqz549KSkpSUlJFhYWCxcunDx5\ncnFxsaGhId66adMmf3//tWvXurq67t27Fz/lcTgc3LsoMjLSysrK3d196dKlbW1tCKGenh4vLy88\nDb2Li8v169fJt4pUi1KtwMDA6upq3JsSMzMzKykpsbOz8/b2trW1vXPnTmFh4bA9OjGBQNDc3FxQ\nUDDUDoN2k5Rx1LDdKpcsWbJr1y47Ozsmk7lmzZr58+eXlZWRiVnlOZe7d+/a2tpOmzZt+NOTn5zj\nimCcJRgt1P9b3bBhg7m5uTprlMg9LrCmpoZGo8kYqkhJX1+fn59fVlaWGo5SidbWVgaDcejQoWH3\nhHGWAKjbCOXdUZ6zs3NSUlJSUpLyM4709fXl5+d3dXVRSoqo2FGqkpCQ4O3tzeVyVVssxE0AxrjY\n2NjVq1eHh4f3+0BEVXFxcV5eXlFRkXSH0BE6SiVSU1MrKiouXbpEqRupPCBuAqCUuLi47Ozsjo4O\nBweHs2fParo5g0tOTuZyufv371emkICAgBMnTpBj7Uf0KOUVFBT09vYWFxePxDSo0O8dAKWkpKSk\npKRouhXDW7x4Mc4LN04EBwcHBwePUOFwvwkAANRA3AQAAGogbgIAADUQNwEAgBoK34XKyspWr149\nck0BQCXKysoQQmP+t4qHD47501SbfqNRZSMk8qWPT01NvXXrlqJNAoCyoqKiGTNmqL//ChjPzpw5\nI89u8sZNANSMIIhTp05JTzIBgJaA95sAAEANxE0AAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGog\nbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANxEwAAqIG4CQAA1EDcBAAAaiBuAgAANRA3AQCA\nGoibAABADcRNAACgBuImAABQA3ETAACogbgJAADUQNwEAABqIG4CAAA1EDcBAIAaiJsAAEANxE0A\nAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGogbgIAADUQNwEAgBqaphsAwP9ob2+XSCTSa7q7u1+9\nekUuGhkZ6enpqb1dAPRH9PulAqApb7755vXr14faqqur++LFC2tra3U2CYBBwXM60BZr164lCGLQ\nTTo6OgsWLICgCbQExE2gLVatWkWjDf7iiCCI9957T83tAWAoEDeBtjAzM1u8eLGuru7ATTo6OqGh\noepvEgCDgrgJtMj69evFYnG/lTQaLTAwkMViaaRJAAwEcRNokaCgIH19/X4r+/r61q9fr5H2ADAo\niJtAizCZzNDQ0H6djQwMDJYuXaqpJgEwEMRNoF3WrVsnFArJRT09vVWrVhkYGGiwSQD0A3ETaJe3\n3npL+lWmUChct26dBtsDwEAQN4F20dPTCw8Pp9PpeNHU1DQgIECzTQKgH4ibQOusXbtWIBAghPT0\n9NavXz9Up04ANAXGWQKtIxaL33jjjaamJoRQaWnp/PnzNd0iAP4PuN8EWkdHR+fdd99FCNnY2Pj4\n+Gi6OQD0N+qfgE6fPq3pJgDVs7CwQAjNnTv3zJkzmm4LUD0fHx87OztNt0Jxo/45fahMEAAArXXq\n1Kk1a9ZouhWKG/X3m2j0/zfQWgRBaPDanj17dtWqVWqoaPXq1QghuLFVmzFwrwPvN4GWUk/QBEAB\nEDcBAIAaiJsAAEANxE0AAKAG4iYAAFADcRMAAKiBuAlU7NKlSywW68KFC5puiJpcuXIlNjYW/1so\nFKakpDg7O9PpdFNTU09Pz/r6+oGH9PT0TJ06ddeuXfKULxaLQ0ND2Ww2g8GwtbUNDg6urKyU58Ck\npCR3d3cTExN9fX1nZ+edO3e+fv2a3Lpv3z7i//L09JQ+fKhzOX/+/GeffdbX1ydPG8YqiJtAxUb7\nSApK9uzZk56eHhcXhxfDwsK+++67EydO8Hi8X3/91cnJSTpUkeLj4x89eiRnFWKxuKSk5OTJk21t\nbaWlpXw+f8GCBY2NjcMeeO3atS1bttTX17e2tqakpKSlpeGeqnIa6lyCgoIYDEZAQEB7e7v8pY01\nklEOIXTq1ClNt2Js0vJry+PxOByO8uWsWrVq1apVChy4f/9+FxcXPp+PF3NycgiCqKyslH3UzZs3\nFy9ejBCKj4+XpxahULhs2TJy8c6dOwih5OTkYQ8MDAwUiUTkIh6/0NDQgBf37t37/fffD3XssOfC\n5XI5HI5QKJTnFPrR8t+VPOB+E4xWWVlZzc3Nmqq9trZ29+7diYmJDAYDrzl69OjMmTO9vLxkHMXn\n83fs2JGWliZ/RTQaTfqlh6OjI0Korq5u2AMvXrwoPTkoHvLP4/HkqXTYc0lISKioqKB0ImMJxE2g\nSqWlpWw2myCIL7/8EiGUmZlpaGjIZDILCgrefvttExMTOzu7nJwcvHN6ejqDwbCystq4caONjQ2D\nwfDx8bl9+zbeyuVy6XT6xIkT8eLmzZsNDQ0JgmhtbUUIRUdHb9++va6ujiAIZ2dnhNC//vUvExOT\n5ORk9Zxpenq6RCIJCgrCiwKBoKyszNvbW/ZR8fHxmzdvtrS0VLhePp+PEDIxMaF64IsXLwwMDBwc\nHIbdU55zMTMzW7hwYVpammQ8vZYhQdwEquTr6/vTTz+Ri5s2bdq6dSufzzc2Nj516lRdXZ2jo+NH\nH32EZxDicrkRERE8Hi8qKqq+vr68vFwkEi1atOjZs2cIofT0dOmh8RkZGYmJieRiWlra8uXLnZyc\nJBJJbW0tQgh/qRg4jfAIKSwsdHV1ZTKZeLGxsVEgENy7d8/f3x//P8DNzS0jI0M6rNy8ebOurk7J\naT/wc7qvry+lo3g83rVr1z766CMykT5CKDY21szMjE6nOzg4hISE3L17V/5zQQjNmDHjxYsXDx48\nUOZ0RimIm0AdfHx8TExMLC0tw8PDu7u7GxoayE00Gs3NzU1fX9/d3T0zM7Orqys7O1uBKgIDAzs7\nO3fv3q26Vg+pu7v7yZMnTk5O5Br8zcTS0jI5Obm6urqpqSkkJGTLli0nT57EO/D5/Ojo6MzMTIUr\nbWpqys3NjYqK4nA45H2unFJSUmxsbPbt20euef/998+fP//s2bPXr1/n5OQ0NDQsXLiwurpannPB\npkyZghCqqqpS+IxGL4ibQK3w/Y70jJXSZs+ezWQyf/vtN/U2irLm5maJRELebCKE8LTvHh4ePj4+\n5ubmLBYrMTGRxWJ9/fXXeIe4uLiPP/7Y1tZW4Uo5HE5UVFRISEhRUVG/qZJlO3fu3OnTpy9fvmxs\nbEyunDRp0owZM4yMjOh0+rx587Kzs/l8fkZGhjznguHTx2n5x5uxkEcOjCX6+votLS2absUwenp6\n0J/xBbOxsUEI4XevGJ1Ot7e3xx9wSktLq6qqUlNTlanUysoqKyvLw8OD0lG5ubmpqanFxcVvvPGG\njN28vLx0dXV///13NNy5kPDkzPhSjDdwvwm0iFAobG9v1/5M4DhkSPf9NjIymjJlysOHD6V3E4lE\neE7jrKysq1ev6ujo4B7m+LtQcnIyQRA///yznJVaWlqamppSaueRI0eOHz9+7do12UETISQWi8Vi\nMf4/gexzIeG588bn1PYQN4EWKS4ulkgk8+bNw4s0Gm2oJ3rNsrKyIgiio6NDemVYWNj9+/cfP36M\nF3k83tOnT3FXnuzsbOnef/iGGvffnD17tpyVXrhwQf7HfIlEEhMTU1VVlZ+fb2RkNHCHt956S3rx\n7t27EomEw+EMey4kfPrW1tZyNmksgbgJNEwsFr969UokElVWVkZHR7PZ7IiICLzJ2dm5ra0tPz9f\nKBS2tLQ8ffpU+kBzc/PGxsb6+vquri6hUFhUVKS2fkhMJtPR0fH58+fSK7dt22Zvbx8REdHQ0PDy\n5cuYmBg+n//JJ5/IU2B4eLi1tXV5eflQO9TW1lpbW4eFhcl51MOHDz///PNjx47p6elJD6Y8dOgQ\n3uHFixe5ubnt7e1CofDWrVsffvghm82OjIyU/1zw6cvurzpWQdwEqvTll1/OmTMHIRQTExMcHJyZ\nmXn48GGE0LRp0x4/fnzs2LHt27cjhJYsWVJTU4MP6enp8fLyMjAw8PPzc3FxuX79OvnecNOmTf7+\n/mvXrnV1dd27dy9+JORwOLijUmRkpJWVlbu7+9KlS9va2tR8poGBgdXV1bg3JWZmZlZSUmJnZ+ft\n7W1ra3vnzp3CwsJhe3RiAoGgubm5oKBgqB0G7SYp46hhu1UuWbJk165ddnZ2TCZzzZo18+fPLysr\nmzBhgvzncvfuXVtb22nTpg1/emOPGscmjQg0+sdsaS01XNsNGzaYm5uPaBXDUmycZU1NDY1GkzFU\nkZK+vj4/P7+srCw1HKUSra2tDAbj0KFDChw7Bv5m4X4TaNgozazj7OyclJSUlJQ0aOYOSvr6+vLz\n87u6usLDw0f6KFVJSEjw9vbmcrnqr1objP24eejQIfwW/6uvvlJ/7Xl5eY6OjvjV0sSJE9evXz/U\nng8ePAgPD3dwcNDX17ewsJg+fTrZSzk8PJyQ6eLFi9IVDdX3OzU1lSAIHR2dqVOn3rhxY0ROeDyJ\njY1dvXp1eHh4vw9EVBUXF+fl5RUVFUl3CB2ho1QiNTW1oqLi0qVLlLqRjimavuFVFpLjnh+/Sjt6\n9Kh6mjSQk5MTi8WSsUNlZSWTyYyKinry5Amfz3/06NHOnTsDAgLw1rCwsB9++AG/wv/3v/+NEAoK\nChIIBN3d3c3NzR999NGFCxfIihBCEydOFAgE/aoQiUT29vYIIbLYYclzbZURGxuLu8FPnjz5zJkz\nI1eRbArnQ8IuX74cExOjwvZoufz8/JSUFOlMS1SN9O9KDcb+/aac+Hy+j4+Ppmo/dOiQqalpWlra\n5MmTGQyGi4sL+RkEIUQQxPz581ksFo1GI9fo6ekxmUxLS8tZs2ZJFzVr1qw//vgjPz+/XxV5eXnK\nDFYZCSkpKb29vRKJ5MmTJ6N31t/FixcfOHBA061Qn+Dg4NjYWOlMS+MQxM3/odmkZC9fvuzo6JD+\nKEyn08nsYTk5OTKexTZs2LBs2TJycdOmTQiho0eP9tstNTUVf8sGAChpPMbNH3/88T/+4z+YTKaJ\niYmXl1dnZ2e/pGRpaWmGhoY6OjqzZs2ytrbW09MzNDScOXOmn5/fpEmTGAyGqanpzp07yQKVz2A2\nZ86c7u7uN9988+bNm0qe3Ztvvunm5nb9+nXpjOI3b97k8Xg4XS4AQEnjLm52d3cHBQWtWrWqra2t\npqbGxcVFIBD0S0oWHR29Y8cOiURy9OjRJ0+e/PHHHwsWLLh//35sbOz9+/fb2tref//9gwcPkhm0\nlM9gtnPnztmzZz948MDX19fDw+Pzzz9XpkPixo0bEULS38G++OKLbdu2KVwgAEDauIub9fX1nZ2d\nHh4eDAbD2to6Ly8P58EelLu7O5PJnDBhwtq1axFCbDbbwsKCyWTiz+Jk2h7lM5gZGBj89NNP//mf\n/zl16tSHDx/GxMS4ubn9+OOPipX2/vvvGxoafvvtt7hX9uPHj+/evatk2kcAAGnc5UNydHS0srJa\nv359VFRURETE5MmT5TkKf/YViUR4EXe/UO3QaT09PS6Xy+Vyb9++feDAgfz8/NWrVz969MjMzIxq\nUSwWa926dceOHcvNzf3ggw8OHz68adMmOp2OEzFQcvjw4TNnzlA9anQpKytDCFGaswyMc+PuftPA\nwODatWu+vr7JycmOjo7h4eHSQ+W0wdy5c//rv/4rMjKypaXl+vXrihWCvw599dVX7e3tZ86cwU/u\nAACVGHf3mwghDw+PCxcutLS0pKamHjhwwMPDQz1Jwvu5cePGvXv3tm7dihBauXLlqVOnyG5GCKF3\n33336NGjcs6iNZC3t/e8efPKyso2bNiwevVqBW5asa1bt0pPVjEmqh5d5QAAIABJREFU4TvNMX9b\nrT0IgtB0E5Q17u43GxsbcWJBS0vL/fv3z5w5s1+eQbW5d++eoaEh/ndvb2+/ZuCv4cokTcC3nGfP\nnsWhGQCgKuMxbm7cuPG3334TCAT3799/+vQpzvbYLykZpTKpZjATCoVNTU3FxcVk3EQIhYaGnj59\nur29vaOjo6Cg4JNPPgkODlYmbq5Zs8bCwiI0NBTPHAsAUBkNj1dSGhpuzNYXX3yBU6saGhquWLGi\nvr7ex8fHzMxMV1f3jTfeiI+PxyPGysvL7e3tDQwMfH19Y2NjcT/zyZMnl5SUHDhwAGe6tra2PnHi\nRG5uLi7QzMwsJydHIpFcunTJ2Nh43759A2s/d+6c9Oxd/Zw7dw7v9sMPP4SFhTk5Oenr69PpdFdX\n14SEhJ6eHumiOjs7FyxYYG5ujhDS0dFxdnZOTk4eWJGFhcWWLVvwyp07d/7000/437t27cJz6uro\n6Li7u5eUlCh/bccGJcdZAqrGwO+KkIzy6Y8Jgjh16tSYfwenEePk2sL7TTUbA7+rcfecDgAASoK4\nCQA1V65ciY2Nxf8WCoUpKSnOzs50Ot3U1NTT07O+vn7gIT09PVOnTt21a5c85YvF4tDQUDabzWAw\nbG1tg4ODKysr5W+eWCw+fPjwoElqSktL58+fz2QybWxsYmJient7yU1JSUnu7u4mJib6+vrOzs47\nd+4cKq9ov3M5f/78Z599NkqTqCoM4iYAFOzZsyc9PT0uLg4vhoWFfffddydOnODxeL/++quTk9Og\n4SY+Pl46XYBsYrG4pKTk5MmTbW1tpaWlfD5/wYIFjY2N8hxbU1OzYMGCbdu2DezBVl1dvXjx4oCA\ngJaWlnPnzv3jH/8gZxNCCF27dm3Lli319fWtra0pKSlpaWlDDQTody5BQUEMBiMgIKC9vV3OExwL\nNP2CVVlo9L9j1lojfW15PB6Hw9F4UfJ/F9q/f7+Liwufz8eLOTk5BEFUVlbKPurmzZs4owqewHJY\nQqFw2bJl5OKdO3cQQtLfAIdSUVGxYsWK48ePe3t7T58+vd/WsLAwBwcHsViMFw8ePEgQxK+//ooX\nAwMDpVNq4pePDQ0Ncp4Ll8vlcDhCoVCeExwDf7Nwvwk0RoW5+9SQBrC2tnb37t2JiYkMBgOvOXr0\n6MyZM2VP6Mjn83fs2JGWliZ/RTQajUwhiBDC3cjq6uqGPXD69Ol5eXnvvPMOObEdSSQSFRYWLly4\nkOxz/vbbb0skEnJOt4sXL0qn1MRJG/rdtMo4l4SEhIqKCkqnOapB3ARKkUgkqampbm5u+vr6ZmZm\nISEhZLoTLpdLp9Nx5yeE0ObNmw0NDQmCaG1tRQj1y92Xnp7OYDCsrKw2btxoY2PDYDB8fHxu376t\nQFFIFZn9BkpPT5dIJEFBQXhRIBCUlZUNO11lfHz85s2bLS0tFa4XjwM2MTFRuASE0OPHj1+/fs1m\ns8k1uNfaUG9OX7x4YWBg4ODgIL1SxrmYmZktXLgwLS1NMsr758gJ4iZQSkJCQmxsbHx8fHNz840b\nN549e+bn59fU1IQQSk9Pl+5rkpGRkZiYSC72y93H5XIjIiJ4PF5UVFR9fX15eblIJFq0aBGe8pdS\nUUgVmf0GKiwsdHV1JRNINzY2CgSCe/fu+fv740Dv5uaWkZEhHThu3rxZV1enZCYq/Jzu6+urTCF/\n/PEHQsjY2Jhcw2AwDAwM8H+pfng83rVr1z766COczgYb9lxmzJjx4sULMrni2AZxEyiOz+enpqau\nWLFi/fr1LBbLy8vrq6++am1t/frrrxUrkEaj4VtXd3f3zMzMrq6u7OxsBcpRPrNfP93d3U+ePJEe\nwoC//1haWiYnJ1dXVzc1NYWEhGzZsuXkyZN4Bz6fHx0dnZmZqXClTU1Nubm5UVFRHA6HvM9VDP50\n3m9yCz09vUGT2qSkpNjY2JDTAiL5zmXKlCkIoaqqKmXaOVpA3ASKq66ufv369ezZs8k1c+bModPp\n5PO1MmbPns1kMsmnfs1qbm6WSCTSs5Xgd4geHh4+Pj7m5uYsFisxMZHFYpH/z4iLi/v444+VmdOJ\nw+FERUWFhIQUFRUpOXMkfidLJkLEBAIBOYcV6dy5c6dPn758+bL0zak854IvzqA3sGPPeMyHBFQF\ndz0xMjKSXmlqatrV1aWS8vX19VtaWlRSlJJ6enrQn7ESs7GxQQjhF6wYnU63t7fHH3BKS0urqqpS\nU1OVqdTKyiorK8vDw0OZQjD8arizs5Ncw+Pxenp68FmQcnNzU1NTi4uL33jjDXKlnOeCQzC+UGMe\n3G8CxZmamiKE+kXJ9vZ2Ozs75QsXCoWqKkp5OChI9+42MjKaMmVKvyxWIpEIpzLIysq6evWqjo4O\nntEef0tJTk4mCOLnn3+Ws1JLS0t8hZXn4OBgbGz89OlTcg1+ESydOObIkSPHjx+/du2adNBEcp8L\nzoo98AZ2TIK4CRTn6elpZGQk/cdz+/ZtgUBATk1Mo9EUzopfXFwskUhwtioli1KelZUVQRAdHR3S\nK8PCwu7fv//48WO8yOPxnj59irslZWdnS3f3w3fNuM+j9GsN2S5cuKCqqZtpNNrSpUtv3LhBfisr\nKioiCAK/NpVIJDExMVVVVfn5+f2eHuQ/F3xxcMqbMQ/iJlAcg8HYvn37uXPnjh8/3tnZWVVVFRkZ\naWNjs2HDBryDs7NzW1tbfn6+UChsaWmRvt9Bg+XuE4vFr169EolElZWV0dHRbDY7IiJCgaKoZvYb\nFpPJdHR0fP78ufTKbdu22dvbR0RENDQ0vHz5MiYmhs/nf/LJJ/IUGB4ebm1tXV5ePtQOtbW11tbW\nYWFhlI6SYffu3U1NTXv27Onu7r5169bBgwcjIiJcXV0RQg8fPvz888+PHTump6dHSDl06JD85eOL\nI7s365gBcRMoZc+ePSkpKUlJSRYWFgsXLpw8ebJ0XtFNmzb5+/uvXbvW1dV17969+CGOw+Hg3kWR\nkZFWVlbu7u5Lly7F83f29PR4eXkZGBj4+fm5uLhcv36dfKVItSiVCwwMrK6ulv4AbWZmVlJSYmdn\n5+3tbWtre+fOncLCwmF7dGICgaC5uZnsdj7QoB0hZR9VVlbm6+v7xhtv3L59+8GDBzY2NvPnz79x\n4wbe6uHhcfny5R9++GHChAkrV67829/+dvToURl1UXX37l1bW1tlMsaOJiM7HGnkodE/Zktrqfna\nbtiwwdzcXG3VkeQcZ1lTU0Oj0b7//nuVVNrX1+fn55eVlaWGo9SgtbWVwWAcOnRInp3HwN8s3G8C\nLaLNaXWcnZ2TkpKSkpKGShQkv76+vvz8/K6urvDw8JE+Sj0SEhK8vb25XK6mG6ImEDcBkFdsbOzq\n1avDw8P7fSCiqri4OC8vr6ioSLpD6AgdpQapqakVFRWXLl1SspPpKAJxE2iFuLi47Ozsjo4OBweH\ns2fParo5Q0pOTuZyufv371emkICAgBMnTpDD7Uf0qJFWUFDQ29tbXFys8JypoxH0ewdaISUlJSUl\nRdOtkMvixYtxLjWAEAoODg4ODtZ0K9QN7jcBAIAaiJsAAEANxE0AAKAG4iYAAFADcRMAAKghJKM8\nrz05XwoAYLQ4deqUdAL/UWfU90M6deqUppsARkRYWFh0dDSHw9F0Q4DqDTq9+ygy6u83wVhFEMRo\nvysBYxW83wQAAGogbgIAADUQNwEAgBqImwAAQA3ETQAAoAbiJgAAUANxEwAAqIG4CQAA1EDcBAAA\naiBuAgAANRA3AQCAGoibAABADcRNAACgBuImAABQA3ETAACogbgJAADUQNwEAABqIG4CAAA1EDcB\nAIAaiJsAAEANxE0AAKAG4iYAAFADcRMAAKiBuAkAANRA3AQAAGogbgIAADUQNwEAgBqImwAAQA3E\nTQAAoAbiJgAAUANxEwAAqIG4CQAA1NA03QAA/kdOTk5XV5f0mitXrrS3t5OLoaGhlpaWam8XAP0R\nEolE020AACGEIiIivv32Wz09PbyIf5kEQSCE+vr6jIyMmpub9fX1NdlEABBC8JwOtMfatWsRQsI/\niUQikUiE/62rq7t69WoImkBLwP0m0BYikcja2rqtrW3QrVevXn3zzTfV3CQABgX3m0Bb0Gi0tWvX\nks/p0iwsLBYuXKj+JgEwKIibQIusXbtWKBT2W6mnp/fuu+/q6upqpEkADATP6UCLSCQSNpv9/Pnz\nfuvv3LkzZ84cjTQJgIHgfhNoEYIg1q9f3+9RfdKkSbNnz9ZUkwAYCOIm0C79HtX19PQiIiJwbyQA\ntAQ8pwOtM3Xq1EePHpGLv/zyi4eHhwbbA0A/cL8JtM67775LPqq7u7tD0ATaBuIm0Drr168XiUQI\nIT09vffff1/TzQGgP3hOB9po9uzZ9+7dIwiivr6ezWZrujkA/B9wvwm00XvvvYcQmjt3LgRNoIVG\nZT6kW7dupaamaroVYAT19PQQBNHb27t69WpNtwWMIA6Hs23bNk23grJReb/57Nmzs2fParoV40hZ\nWVlZWZk6a2QwGNbW1nZ2duqs9Pnz5/C7UqeysrJbt25puhWKGJX3m9iZM2c03YTxAt/0qfmC19bW\nOjs7q7PG06dPh4WFwe9KbUbvw8SovN8E44GagyYA8oO4CQAA1EDcBAAAaiBuAgAANRA3AQCAGoib\nYKRcunSJxWJduHBB0w0ZKVeuXImNjcX/FgqFKSkpzs7OdDrd1NTU09Ozvr5+4CE9PT1Tp07dtWuX\nPOWLxeLQ0FA2m81gMGxtbYODgysrK+VvnlgsPnz4sI+Pz8BNpaWl8+fPZzKZNjb/v71zD2rq+B74\nXkzz4hGCEkQQ5CUKgmhtSxBEh5FWaYX6Aq0zpY4v1AlUaxEiCqlBrRYzVLBTm6ZjfSA+BlREO1VR\n8V0VofgEBFQqAUFeCSQh9/fHjvnmhxJuAglJ2M9f7t7ds2cz18Pe3bPnOCYmJnZ1dake8Xg8Hx8f\nGxsbCoXi6en5/ffft7e3v1d+j7mcPHlyx44d3d3dxDU0XZDdROgL877Cu2XLlszMzOTkZFiMjo7e\nv3//wYMHJRLJw4cPPTw83mtuuFyueqgnzSiVyitXrhw6dKipqam4uFgqlU6bNq2uro5I36dPn06b\nNm3dunUSiaTHo/Ly8vDw8LCwsIaGhhMnTvz+++9xcXGqpxcuXFi7dm11dXVjY2N6erpAIOjNW6jH\nXObMmUOlUsPCwtRTN5stuAly5MgRE9XcRJk/f/78+fMHW4tekUgkbDa7/3KIv1fbtm0bO3asVCqF\nxcOHD2MYVlpaqrnX1atXw8PDAQBcLpfIKHK5/PPPP1cVb926BQDg8/l9diwpKZk7d+6BAwcCAgIm\nTpzY42l0dLSbm5tSqYTFnTt3Yhj28OFDWIyIiFAoFKrGCxcuBADU1tYSnAuHw2Gz2XK5nMgEjfy9\n0gBabyJMHqFQKBaLDTZcRUVFSkpKWloalUqFNXv37p08ebKfn5+GXlKpdMOGDQKBgPhAJBJJfZfD\n3d0dAFBZWdlnx4kTJx4/fvyrr756N3OyQqEoKCgIDQ1VhYKeNWsWjuP5+fmwePr0afVUTiNGjAAA\n9Fi0aphLampqSUmJVtM0RZDdROiF4uJiFxcXDMP27NkDAMjOzra0tKTT6fn5+bNmzbKxsXF2dj58\n+DBsnJmZSaVSWSzWqlWrHB0dqVRqUFDQzZs34VMOh0Mmk0eOHAmLa9assbS0xDCssbERAJCQkLB+\n/frKykoMw6Cr/NmzZ21sbPh8vp6mlpmZieP4nDlzYFEmk924cSMgIEBzLy6Xu2bNGnt7e53HlUql\nAAAbGxudJQAAqqqq2tvb1aOleHh4AAB62zl9+fIljUZzc3NTr9QwFyaTGRoaKhAIcLPepUF2E6EX\ngoODr127piquXr3622+/lUql1tbWR44cqaysdHd3X758OUyJweFwYmNjJRJJfHx8dXX13bt3FQrF\nzJkznz9/DgDIzMyEX4uQrKystLQ0VVEgEHzxxRceHh44jldUVAAA4NGEUqnU09QKCgq8vb3pdDos\n1tXVyWSyO3fuzJgxAxr98ePHZ2VlqRuOq1evVlZWLl68uD/jwu/04ODg/gh59eoVAMDa2lpVQ6VS\naTRafX39u40lEsmFCxeWL19OJpNVlX3OZdKkSS9fvrx//35/9DRykN1EGJSgoCAbGxt7e/uYmJiO\njo7a2lrVIxKJNH78eAqF4uPjk52d3dbWJhKJdBgiIiKitbU1JSVl4LT+Hx0dHc+ePYNrNAg8/7G3\nt+fz+eXl5fX19VFRUWvXrj106BBsIJVKExISsrOzdR60vr4+JycnPj6ezWar1rm6AY/OeyRV/uCD\nD+Bitgfp6emOjo5bt25V1RCZi5eXFwCgrKysP3oaOchuIgYHuIR5N1s6ZMqUKXQ6/dGjR4ZVqm/E\nYjGO46rFJgAA7iH6+voGBQXZ2dkxGIy0tDQGg/Hrr7/CBsnJyStWrHByctJ5UDabHR8fHxUVVVhY\n2CPZp7bAPVkYTl+FTCaj0Wg9Wp44cSI3N/fcuXPqi1Mic4E/znsXsGaDCcdDQpg3FAqloaFhsLXo\nSWdnJ3hrKyGOjo4AALjZCiGTya6urvAAp7i4uKysrJ/hYlksllAoHJA8S3CbuLW1VVUjkUg6Ozvh\nLFTk5ORkZGQUFRWNGjVKVUlwLtAEwx/KXEHrTYQxIpfL37x5Y+D4m0SARkHdu9vKysrLy+vBgwfq\nzRQKBYPBAAAIhcLz589bWFhgGIZhGDxL4fP5GIb9888/BAe1t7e3tbUdEP3d3Nysra1rampUNXBT\n2N/fX1Xz888/Hzhw4MKFC+pGExCei0wmA29/KHMF2U2EMVJUVITjeGBgICySSKTevugNDIvFwjCs\npaVFvTI6OvrevXtVVVWwKJFIampqoFuSSCRS9/uDK2jo8zhlyhSCg546dao/n/nqkEik2bNnX758\nWXVuVlhYiGEY3DbFcTwxMbGsrCwvL8/KyqpHX4JzgT+Og4PDgChsnCC7iTAWlEplc3OzQqEoLS1N\nSEhwcXGJjY2Fjzw9PZuamvLy8uRyeUNDg/pyCQBgZ2dXV1dXXV3d1tYml8sLCwv154dEp9Pd3d1f\nvHihXrlu3TpXV9fY2Nja2trXr18nJiZKpdKNGzcSERgTE+Pg4HD37t3eGlRUVDg4OERHR2vVSwMp\nKSn19fVbtmzp6Oi4fv36zp07Y2Njvb29AQAPHjz48ccf9+3b98EHH2Bq7Nq1i7h8+ONo9mY1dZDd\nROiFPXv2fPTRRwCAxMTEyMjI7Ozs3bt3AwD8/f2rqqr27du3fv16AMBnn3329OlT2KWzs9PPz49G\no4WEhIwdO/bixYuqbcTVq1fPmDFj0aJF3t7eP/zwA/wGZLPZ0FEpLi6OxWL5+PjMnj27qalJ31OL\niIgoLy9XP4BmMplXrlxxdnYOCAhwcnK6detWQUFBnx6dEJlMJhaLVW7n7/JeR0jNvW7cuBEcHDxq\n1KibN2/ev3/f0dFx6tSply9fhk99fX3PnTv3119/DR8+fN68eUuXLt27d6+GsbTl9u3bTk5O6h/+\nZoi+LyTpA3TP0sAY4D7cypUr7ezs9DpEnxB8r54+fUoikf78888BGbS7uzskJEQoFBqglwFobGyk\nUqm7du0i0hjds0Qg+ouphNLx9PTk8Xg8Hq+3QEHE6e7uzsvLa2tri4mJ0Xcvw5CamhoQEMDhcAZb\nEf2C7CYCoTVJSUkLFiyIiYnpcUCkLUVFRcePHy8sLFR3CNVTLwOQkZFRUlJy5syZfjqZGj9DxW4u\nW7bM2toaw7CSkpLB1gUAAI4fP+7u7q6+9U4mk1ks1vTp03fu3Nnc3DzYChqU5ORkkUjU0tLi5uZm\nKpl4+Xw+h8PZtm1bf4SEhYUdPHhQdfVer730TX5+fldXV1FREZPJHGxd9M5QsZu//fbbvn37BluL\n/zFv3ryqqioPDw8Gg4HjuFKpFIvFubm5bm5uiYmJvr6+xJ37zID09PSuri4cx589ezZ//vzBVoco\n4eHh27dvH2wtjIXIyMikpKQeNzjNlaFiN40cDMNsbW2nT58uEolyc3Pr6+sjIiL6+Q2IQCD0xBCy\nm6qAg0bO/PnzY2NjxWLxL7/8Mti6IBCI92DOdhPH8Z07d3p7e1MoFAaDsWHDBvWn3d3dmzdvdnFx\nodFo/v7+0AdFc5hIAMClS5c+/vhjOp1uY2Pj5+cH7/m+VxToRyBI6O9dWFhoMFURCIQWDLIflE4Q\n9LPjcrkYhv3000/Nzc0SiSQrKwsAcO/ePfj0u+++o1Aox44da25uTk5OtrCwuH37NuwFADh//nxL\nS4tYLA4JCbG0tJTJZDiOt7e329jY7NixQyqVvnr1au7cuQ0NDRpEnT592tramsfj9aahan+zB9DG\njR492mCqasZ0/ey0AvkFGxjTfa9M8i0h8n5LJBI6nT5z5kxVDVyLQbsplUrpdHpMTIyqMYVCWb16\nNf7WGKlSx0BrW1FRgeP4v//+CwA4ffq0+kAaRPVJb3YTx3G442kkqpru+60VyG4aGNN9r8w2jlxF\nRYVEIgkLC3vv08ePH0skkgkTJsAijUYbOXLke6M9qoeJdHd3Z7FYS5YsiY+Pj42NHTNmjFaiiNPR\n0YHjOMyIYCSqHjt2zFQ2iPvJEJmmkWBC7hPqmK3dhMEFekvn0tHRAQDYtGmTeibrHiEI34VGo124\ncGHjxo18Pp/H4y1cuFAkEukmSjNPnjwBAIwbN854VA0MDPz222+1n4opcf36dYFAgPZ8DQYMWWCK\nmK3dhHGtYVaAd4H2dPfu3QkJCVqJ9fX1PXXqVENDQ0ZGxvbt2319feFdNx1EaeDs2bMAgFmzZhmP\nqs7OzupJfswVgUAwFKZpJBw9enSwVdARsz1PnzBhgoWFxaVLl977dPTo0VQqVdu7Q3V1dTA8rb29\n/bZt2yZPnvzgwQPdRGng1atXu3fvdnZ2Xrp0qZGrikAMTczWbtrb28+bN+/YsWNCobC1tbW0tFSV\n7wUAQKVSv/nmm8OHD2dnZ7e2tnZ3d7948eK///7TLLOurm7VqlWPHj2SyWT37t2rqakJDAzUIIpI\nIEgcx9vb25VKJY7jDQ0NR44cmTp16rBhw/Ly8uD+pmFURSAQWjDI51I6QfDcs62tbdmyZcOHD7ey\nsgoODt68eTMAwNnZ+f79+ziOd3V1JSYmuri4kEgkaGTLy8uzsrJgrAQvL6/Kyspff/0VGi9XV9cn\nT55UV1cHBQUxmcxhw4aNGjWKy+UqFIreROE4fubMGWtr661bt76r28mTJ/39/el0OplMtrCwAG+v\nDH388cc8Hu/169fqjQ2gqmZM99xTK9B5uoEx3fcKw00wPXxubm50dLQpam6iLFiwAJjybhRB0Htl\nYEz3vTLb73QEAoHQE8huIhB64e+//05KSoL/lsvl6enpnp6eZDLZ1tZ2woQJ1dXV73bp7OwcN26c\nupeYBpRK5Zdffuni4kKlUp2cnCIjI0tLSwnqpkGfrVu3Yv8flcPvyZMnd+zYYSrhpfUKspsIxMCz\nZcuWzMzM5ORkWIyOjt6/f//BgwclEsnDhw89PDzeGyuey+U+fvyY4BBKpfLKlSuHDh1qamoqLi6W\nSqXTpk2rq6sj0pegPj2YM2cOlUoNCwt78+YNQSXNFWQ3EUaBVCoNCgoyNlG6sX379pycnNzcXGtr\nawBATk5OXl7e0aNHP/nkExKJ5OjomJ+fr1rEqbh27Rq8HUscNpsdHBxMp9Pd3Nz4fH5LS8sff/zR\nZ68+9emROkldq/j4+IkTJ86ePVuhUGilqpmB7CbCKBAKhWKx2NhE6UBFRUVKSkpaWhq8eQEA2Lt3\n7+TJkzXnxZVKpRs2bBAIBMQHIpFIp06dUhXd3d0BAJWVlX12JKKPBlJTU0tKSrRS1fxAdhMxYOA4\nnpGRMX78eAqFwmQyo6KiVJffORwOmUxWpXZYs2aNpaUlhmGNjY0AgISEhPXr11dWVmIY5unpmZmZ\nSaVSWSzWqlWrHB0dqVRqUFDQzZs3dRAF+hHNTzcyMzNxHJ8zZw4symSyGzdu9JkTmMvlrlmzprdr\nwUSAeYmhL5oGCOqjASaTGRoaKhAIhrLjAbKbiAEjNTU1KSmJy+WKxeLLly8/f/48JCSkvr4eAJCZ\nmal+fzErKystLU1VFAgEX3zxhYeHB47jFRUVHA4nNjZWIpHEx8dXV1ffvXtXoVDMnDkTZkvXShR4\nmyZTqVTq/wcAAICCggJvb29VxrS6ujqZTHbnzp0ZM2bAvwHjx4/PyspSNzpXr16trKxcvHhxf8a9\ndesWACA4OFhzMyL6JCUlMZlMMpns5uYWFRV1+/btHkImTZr08uXL+/fv90dhkwbZTcTAIJVKMzIy\n5s6du2TJEgaD4efn98svvzQ2Nqpf09IKEokEl64+Pj7Z2dltbW0ikUgHOREREa2trSkpKbqpoRUd\nHR3Pnj3z8PBQ1cDzFnt7ez6fX15eXl9fHxUVtXbt2kOHDsEGUqk0ISEhOztb50Hr6+tzcnLi4+PZ\nbLZqndsbferz9ddfnzx58vnz5+3t7YcPH66trQ0NDS0vL1cX4uXlBQAoKyvTWWdTB9lNxMBQXl7e\n3t4+ZcoUVc1HH31EJpNV39f9YcqUKXQ6vZ/R+QyAWCzGcVw9PS+FQgEA+Pr6BgUF2dnZMRiMtLQ0\nBoOh+nOSnJy8YsUKJycnnQdls9nx8fFRUVGFhYV9JuDtU5/Ro0dPmjTJysqKTCYHBgaKRCKpVApD\nu6qAE4RfEkMTs42HhDAw0DfFyspKvdLW1ratrW1A5FMolIaGhgERpT86OzvBW9sEgWH64N4rhEwm\nu7q6wgOc4uLisrKyjIyM/gzKYrGEQqGvry+Rxpr1eRc/P79hw4bBwIYqaDQaeDvZoQlabyIGBltb\nWwBADyv55s0bZ2fn/guXy+UDJUqvQIOi7hluZWXl5eUFY1N8YA+sAAADvklEQVSpUCgUDAYDACAU\nCs+fP29hYQE9zOG5EJ/PxzCMeCJoe3t7+OMTQbM+76JUKpVKpfpfAgCATCYDbyc7NEF2EzEwTJgw\nwcrKSv1/+82bN2Uy2YcffgiLJBIJxqLXgaKiIhzHAwMD+y9Kr7BYLAzDeiRwjo6OvnfvXlVVFSxK\nJJKamhroBiQSidQ9JeGCmsvl4jiuvuOhmVOnTmn1ma9BHwDAp59+qt4Ypp9is9nqlXCCDg4OxAc1\nM5DdRAwMVCp1/fr1J06cOHDgQGtra1lZWVxcnKOj48qVK2EDT0/PpqamvLw8uVze0NBQU1Oj3t3O\nzq6urq66urqtrQ3aRKVS2dzcrFAoSktLExISXFxcYJpPbUURieY3UNDpdHd3d5hrQMW6detcXV1j\nY2Nra2tfv36dmJgolUo3btxIRGBMTIyDg8Pdu3d7a1BRUeHg4BAdHU28l2Z9Xr58mZOT8+bNG7lc\nfv369WXLlrm4uMTFxalLgBPU2QPUDEB2EzFgbNmyJT09ncfjjRgxIjQ0dMyYMUVFRZaWlvDp6tWr\nZ8yYsWjRIm9v7x9++AF+5bHZbOhdFBcXx2KxfHx8Zs+e3dTUBADo7Oz08/Oj0WghISFjx469ePGi\n6mtRW1GGJCIiory8HHpTQphM5pUrV5ydnQMCApycnG7dulVQUEDQg1Imk4nF4vz8/N4avNeJUnMv\nzfp89tlnmzZtcnZ2ptPpCxcunDp16o0bN4YPH64u4fbt205OTv7+/kSmYJ7oN0ydfkBxEg2M4eMk\nrly50s7OzpAj4gP0Xj19+pREIvW4qqgz3d3dISEhQqHQAL0I0tjYSKVSd+3a1X9Rpht/E603EUaK\nicbd8fT05PF4PB6PSKQMzXR3d+fl5bW1tcHEUHrtRZzU1NSAgAAOh6MP4aYCspsIxACTlJS0YMGC\nmJiYHgdE2lJUVHT8+PHCwkJ1h1A99SJIRkZGSUnJmTNn+nQUNW+Q3UQYHcnJySKRqKWlxc3N7dix\nY4Otji7w+XwOh7Nt27b+CAkLCzt48KDqJr5eexEhPz+/q6urqKiIyWQOuHDTAvm9I4yO9PT09PT0\nwdaiv4SHh4eHhw+2FgNJZGRkZGTkYGthFKD1JgKBQGgHspsIBAKhHchuIhAIhHYgu4lAIBDaYcLn\nQrm5uYOtwlAB3qsz+x/8+vXrYAhM03h48eKF8cdqeT+D7XivC/BeBwKBMHVM9L4Qhg/hJCEIBAKh\nA2h/E4FAILQD2U0EAoHQDmQ3EQgEQjuQ3UQgEAjt+D+k2lIo1gYIxwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 28,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.keras.utils.plot_model(\n",
    "    model,\n",
    "    show_shapes=True,\n",
    "    show_layer_names=True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JpL9idwZV6fL"
   },
   "source": [
    "For each character the model looks up the embedding, runs the GRU one timestep with the embedding as input, and applies the dense layer to generate logits predicting the log-likelihood of the next character:\n",
    "\n",
    "![Model architecture](https://www.tensorflow.org/tutorials/text/images/text_generation_training.png)\n",
    "\n",
    "Image source: [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation) notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Npruiy2RAPkt"
   },
   "source": [
    "## Try the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11144,
     "status": "ok",
     "timestamp": 1584519814917,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "E4DCLA0GASL1",
    "outputId": "eba67358-d6b9-4b42-a2f1-dc6ed3e0708c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(64, 100, 65) # (batch_size, sequence_length, vocab_size)\n"
     ]
    }
   ],
   "source": [
    "for input_example_batch, target_example_batch in dataset.take(1):\n",
    "    example_batch_predictions = model(input_example_batch)\n",
    "    print(example_batch_predictions.shape, \"# (batch_size, sequence_length, vocab_size)\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MWebJXU9CEPd"
   },
   "source": [
    "To get actual predictions from the model we need to sample from the output distribution, to get actual character indices. This distribution is defined by the logits over the character vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 340
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11020,
     "status": "ok",
     "timestamp": 1584519814919,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Y4Jgo-iECFWI",
    "outputId": "a59cb406-3813-4fb0-8297-6862a73f2499"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction for the 1st letter of the batch 1st sequense:\n",
      "tf.Tensor(\n",
      "[-5.5658985e-03 -5.6167855e-03  2.3333444e-03 -5.4010577e-03\n",
      " -1.2658490e-03 -2.0685978e-03 -1.7119508e-03 -1.9059415e-03\n",
      " -1.8099162e-03  1.8549486e-03 -6.1538210e-03  2.1649192e-03\n",
      "  5.9574037e-03  3.1515316e-03 -3.3867492e-03  6.0820174e-03\n",
      "  2.0375988e-03  1.1445739e-03  1.3204901e-03 -2.5559508e-03\n",
      " -6.7244383e-04  2.0627938e-03 -5.1490301e-03  5.0887186e-03\n",
      " -2.1955089e-03 -5.3793588e-03  3.5228319e-03 -1.3459967e-03\n",
      "  1.5698730e-03 -3.6733383e-03 -2.6844436e-04  1.2893401e-03\n",
      " -2.9499468e-03 -2.0524827e-03  4.9254433e-03 -1.4868007e-03\n",
      " -3.2468680e-03 -3.1228375e-03  1.4175450e-03  2.6028175e-03\n",
      " -1.9702244e-04 -1.5901095e-04 -5.2650859e-05 -2.0922027e-03\n",
      " -1.1370353e-03 -1.3396480e-03  2.7639134e-04  2.6140050e-03\n",
      "  3.0784088e-04 -7.5585191e-04  2.4966525e-03  1.0233605e-03\n",
      "  1.4258339e-03 -1.6010429e-04  5.1720534e-03  9.4948718e-03\n",
      " -4.6309843e-03  5.5265515e-03 -1.2462666e-04 -1.3864412e-03\n",
      " -2.6601439e-03  2.0305226e-03  1.0555825e-03  1.6011465e-03\n",
      "  2.9543908e-03], shape=(65,), dtype=float32)\n"
     ]
    }
   ],
   "source": [
    "print('Prediction for the 1st letter of the batch 1st sequense:')\n",
    "print(example_batch_predictions[0, 0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10898,
     "status": "ok",
     "timestamp": 1584519814922,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "0dOr0MwFHlRb",
    "outputId": "f8dc2264-a19a-4887-f964-9086a95b14a4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor([[2 2 2 1 2]], shape=(1, 5), dtype=int64)\n"
     ]
    }
   ],
   "source": [
    "# Quick overview of how tf.random.categorical() works.\n",
    "\n",
    "# logits is 2-D Tensor with shape [batch_size, num_classes].\n",
    "# Each slice [i, :] represents the unnormalized log-probabilities for all classes.\n",
    "# In the example below we say that the probability for class \"0\" is low but the\n",
    "# probability for class \"2\" is much higher.\n",
    "tmp_logits = [\n",
    "  [-0.95, 0, 0.95],\n",
    "];\n",
    "\n",
    "# Let's generate 5 samples. Each sample is a class index. Class probabilities \n",
    "# are being taken into account (we expect to see more samples of class \"2\").\n",
    "tmp_samples = tf.random.categorical(\n",
    "    logits=tmp_logits,\n",
    "    num_samples=5\n",
    ")\n",
    "\n",
    "print(tmp_samples)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10502,
     "status": "ok",
     "timestamp": 1584519814922,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "JPzr0r4zCgS3",
    "outputId": "23e7516b-1753-402a-bc61-8a8988e49f18"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([100, 1])"
      ]
     },
     "execution_count": 32,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices = tf.random.categorical(\n",
    "    logits=example_batch_predictions[0],\n",
    "    num_samples=1\n",
    ")\n",
    "\n",
    "sampled_indices.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10276,
     "status": "ok",
     "timestamp": 1584519814923,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "YaA7DclID8dz",
    "outputId": "08a3c58b-e331-4f91-8e3f-be2649a85f48"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100,)"
      ]
     },
     "execution_count": 33,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices = tf.squeeze(\n",
    "    input=sampled_indices,\n",
    "    axis=-1\n",
    ").numpy()\n",
    "\n",
    "sampled_indices.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 119
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 10212,
     "status": "ok",
     "timestamp": 1584519814924,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "_ubGQ0gVENhB",
    "outputId": "4ab910cf-06e6-4fd1-c751-746d79d895d0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([62, 61, 40, 20,  4, 44, 17, 43, 28, 18, 50, 48,  0, 11, 58, 17, 41,\n",
       "       22, 62, 46, 12, 24, 25, 54, 10,  6, 18,  5,  4, 23, 55, 51, 38,  9,\n",
       "       50, 42, 42,  9, 49,  4, 48, 54, 59,  4, 30,  7, 20,  4, 53, 27, 21,\n",
       "        5, 16, 27, 25, 47, 21,  7, 24, 25, 46, 21,  6, 56,  1, 21, 24, 28,\n",
       "       55,  8, 23,  0, 45, 29,  4, 29,  5, 49, 25, 45,  3,  4, 22, 11, 13,\n",
       "       30, 43,  0, 62,  0, 48, 29, 33, 58, 46, 51, 12, 59, 55, 62])"
      ]
     },
     "execution_count": 34,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 122
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9992,
     "status": "ok",
     "timestamp": 1584519814925,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Gi9HOzw9EajS",
    "outputId": "aa7dc69f-63ad-4273-b2df-e6cc8eada7cd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input:\n",
      " 'is service.\\n\\nBAPTISTA:\\nA thousand thanks, Signior Gremio.\\nWelcome, good Cambio.\\nBut, gentle sir, met'\n",
      "\n",
      "Next char prediction:\n",
      " \"xwbH&fEePFlj\\n;tEcJxh?LMp:,F'&KqmZ3ldd3k&jpu&R-H&oOI'DOMiI-LMhI,r ILPq.K\\ngQ&Q'kMg$&J;ARe\\nx\\njQUthm?uqx\"\n"
     ]
    }
   ],
   "source": [
    "print('Input:\\n', repr(''.join(index2char[input_example_batch[0]])))\n",
    "print()\n",
    "print('Next char prediction:\\n', repr(''.join(index2char[sampled_indices])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9825,
     "status": "ok",
     "timestamp": 1584519814926,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "b87e0lsYMTsv",
    "outputId": "75c7e1bf-b2d5-4f15-cba7-0ae5f8765327"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction  0\n",
      "  input: 47 ('i')\n",
      "  next predicted: 1 ('x')\n",
      "Prediction  1\n",
      "  input: 57 ('s')\n",
      "  next predicted: 1 ('w')\n",
      "Prediction  2\n",
      "  input: 1 (' ')\n",
      "  next predicted: 1 ('b')\n",
      "Prediction  3\n",
      "  input: 57 ('s')\n",
      "  next predicted: 1 ('H')\n",
      "Prediction  4\n",
      "  input: 43 ('e')\n",
      "  next predicted: 1 ('&')\n"
     ]
    }
   ],
   "source": [
    "for i, (input_idx, sample_idx) in enumerate(zip(input_example_batch[0][:5], sampled_indices[:5])):\n",
    "    print('Prediction {:2d}'.format(i))\n",
    "    print('  input: {} ({:s})'.format(input_idx, repr(index2char[input_idx])))\n",
    "    print('  next predicted: {} ({:s})'.format(target_idx, repr(index2char[sample_idx])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LqcBufKEE_p6"
   },
   "source": [
    "## Train the model\n",
    "\n",
    "At this point the problem can be treated as a standard classification problem. Given the previous RNN state, and the input this time step, predict the class of the next character."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "l4s0-PvrFub5"
   },
   "source": [
    "### Attach an optimizer, and a loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9644,
     "status": "ok",
     "timestamp": 1584519814927,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "UOEUUm6JE95a",
    "outputId": "ba94797c-e3c6-4d71-ac44-2cf76f7b2843"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction shape:  (64, 100, 65)  # (batch_size, sequence_length, vocab_size)\n",
      "scalar_loss:       4.17424\n"
     ]
    }
   ],
   "source": [
    "# An objective function.\n",
    "# The function is any callable with the signature scalar_loss = fn(y_true, y_pred).\n",
    "def loss(labels, logits):\n",
    "    return tf.keras.losses.sparse_categorical_crossentropy(\n",
    "      y_true=labels,\n",
    "      y_pred=logits,\n",
    "      from_logits=True\n",
    "    )\n",
    "\n",
    "example_batch_loss = loss(target_example_batch, example_batch_predictions)\n",
    "\n",
    "print(\"Prediction shape: \", example_batch_predictions.shape, \" # (batch_size, sequence_length, vocab_size)\")\n",
    "print(\"scalar_loss:      \", example_batch_loss.numpy().mean())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SXhJsB6eFgrJ"
   },
   "outputs": [],
   "source": [
    "adam_optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)\n",
    "model.compile(\n",
    "    optimizer=adam_optimizer,\n",
    "    loss=loss\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MK3Cf-xZFwL4"
   },
   "source": [
    "### Configure checkpoints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LUhXnHPJFy5q"
   },
   "outputs": [],
   "source": [
    "# Directory where the checkpoints will be saved.\n",
    "checkpoint_dir = 'tmp/checkpoints'\n",
    "os.makedirs(checkpoint_dir, exist_ok=True)\n",
    "\n",
    "# Name of the checkpoint files\n",
    "checkpoint_prefix = os.path.join(checkpoint_dir, 'ckpt_{epoch}')\n",
    "\n",
    "checkpoint_callback=tf.keras.callbacks.ModelCheckpoint(\n",
    "    filepath=checkpoint_prefix,\n",
    "    save_weights_only=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "oFg9MFJoGZWf"
   },
   "source": [
    "### Execute the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "AVk-pARPGaja"
   },
   "outputs": [],
   "source": [
    "EPOCHS=40"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1197604,
     "status": "ok",
     "timestamp": 1584521003714,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "y0rveBdAGeEz",
    "outputId": "2449548c-5ca4-4340-df15-9cdf1714dcbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train for 172 steps\n",
      "Epoch 1/40\n",
      "172/172 [==============================] - 31s 179ms/step - loss: 2.5607\n",
      "Epoch 2/40\n",
      "172/172 [==============================] - 29s 171ms/step - loss: 1.8654\n",
      "Epoch 3/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 1.6191\n",
      "Epoch 4/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 1.4893\n",
      "Epoch 5/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 1.4108\n",
      "Epoch 6/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 1.3550\n",
      "Epoch 7/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 1.3101\n",
      "Epoch 8/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 1.2705\n",
      "Epoch 9/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 1.2335\n",
      "Epoch 10/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 1.1980\n",
      "Epoch 11/40\n",
      "172/172 [==============================] - 29s 171ms/step - loss: 1.1610\n",
      "Epoch 12/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 1.1230\n",
      "Epoch 13/40\n",
      "172/172 [==============================] - 30s 174ms/step - loss: 1.0843\n",
      "Epoch 14/40\n",
      "172/172 [==============================] - 30s 174ms/step - loss: 1.0446\n",
      "Epoch 15/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 1.0031\n",
      "Epoch 16/40\n",
      "172/172 [==============================] - 30s 174ms/step - loss: 0.9617\n",
      "Epoch 17/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 0.9211\n",
      "Epoch 18/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 0.8810\n",
      "Epoch 19/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 0.8413\n",
      "Epoch 20/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.8018\n",
      "Epoch 21/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.7692\n",
      "Epoch 22/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.7346\n",
      "Epoch 23/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 0.7056\n",
      "Epoch 24/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.6777\n",
      "Epoch 25/40\n",
      "172/172 [==============================] - 30s 174ms/step - loss: 0.6528\n",
      "Epoch 26/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.6282\n",
      "Epoch 27/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 0.6077\n",
      "Epoch 28/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.5897\n",
      "Epoch 29/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 0.5735\n",
      "Epoch 30/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.5566\n",
      "Epoch 31/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.5447\n",
      "Epoch 32/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.5318\n",
      "Epoch 33/40\n",
      "172/172 [==============================] - 30s 174ms/step - loss: 0.5198\n",
      "Epoch 34/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.5103\n",
      "Epoch 35/40\n",
      "172/172 [==============================] - 30s 172ms/step - loss: 0.4999\n",
      "Epoch 36/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.4930\n",
      "Epoch 37/40\n",
      "172/172 [==============================] - 30s 174ms/step - loss: 0.4858\n",
      "Epoch 38/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.4782\n",
      "Epoch 39/40\n",
      "172/172 [==============================] - 30s 173ms/step - loss: 0.4719\n",
      "Epoch 40/40\n",
      "172/172 [==============================] - 29s 170ms/step - loss: 0.4672\n"
     ]
    }
   ],
   "source": [
    "history = model.fit(\n",
    "  x=dataset,\n",
    "  epochs=EPOCHS,\n",
    "  callbacks=[\n",
    "    checkpoint_callback\n",
    "  ]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "mLdnOyvzMggJ"
   },
   "outputs": [],
   "source": [
    "def render_training_history(training_history):\n",
    "    loss = training_history.history['loss']\n",
    "    plt.title('Loss')\n",
    "    plt.xlabel('Epoch')\n",
    "    plt.ylabel('Loss')\n",
    "    plt.plot(loss, label='Training set')\n",
    "    plt.legend()\n",
    "    plt.grid(linestyle='--', linewidth=1, alpha=0.5)\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1197218,
     "status": "ok",
     "timestamp": 1584521003724,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "4Ghveem_OQBV",
    "outputId": "f4fcc0aa-4509-4330-a88b-1f5863c354d2"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjAsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8GearUAAAgAElEQVR4nO3deXxU5b348c+TzCQTMiFDMkA2SCAs\nYQ+CIK6AWlvcamvrdq1t7RWt1i62bl20/dVq19ta7bVaW7XutvVq1VoVUBFUCgqyRQkwQAIBBpyQ\nQCbr8/tjJjFAEjInmTlP5nzfr1dezJxZ8jlnSJ7MOWfOUVprhBBCOFeK3QFCCCHsJQOBEEI4nAwE\nQgjhcDIQCCGEw8lAIIQQDicDgRBCOJwMBEII4XAyEAjRA6VUQCl1ht0dQsSTDARCCOFwMhAIESOl\nVLpS6rdKqZ3Rr98qpdKjt/mVUi8opUJKqf1KqaVKqZTobTcppaqVUnVKqQ+VUqfbOydCRLjsDhBi\nAPo+cAJQDmjgOeAHwA+BG4AqYGj0vicAWik1HrgOOF5rvVMpVQKkJjZbiK7JOwIhYncZ8BOt9R6t\n9V7gx8Dl0duagXygWGvdrLVeqiMH9GoF0oGJSim31jqgtd5sS70QR5CBQIjYFQDbOl3fFp0G8Eug\nEnhFKbVFKXUzgNa6EvgWcDuwRyn1pFKqACEMIAOBELHbCRR3uj4yOg2tdZ3W+gat9WjgPOA77dsC\ntNaPa61Pjj5WAz9PbLYQXZOBQIhjcyulPO1fwBPAD5RSQ5VSfuBHwKMASqlzlFJjlFIKqCWySqhN\nKTVeKTU/ulE5DDQAbfbMjhCHk4FAiGN7icgv7vYvD7AS+ABYC7wH/DR637HAa0A98DbwB631EiLb\nB+4CgkANMAy4JXGzIET3lJyYRgghnE3eEQghhMPJQCCEEA4nA4EQQjicDARCCOFwA+4QE36/X5eU\nlFh6bEtLCy6XmbMsbdaY3AZm90mbNQO1bdWqVUGt9dCubjNzbnpQUlLCypUrLT22oaGBjIyMfi7q\nH9JmjcltYHaftFkzUNuUUtu6vAFZNSSEEI7nqIFg27ZuB0TbSZs1JreB2X3SZk0ytjlqIBBCCHG0\nAbeNQAgxcDQ3N1NVVUU4HI75cRs3boxTVd+Y3rZ161aKiopwu929fpyjBgK/3293QrekzRqT28Ds\nvkS0VVVVkZWVRUlJCZHj8PVOc3NzTL/IEsnktqamJg4cOEBVVRWjRo3q9eMctWrI6T+UVkmbdSb3\nJaItHA6Tm5sb0yAAGPuLFsxuS0tLIzc3N+Z3YI4aCCorK+1O6Ja0WWNyG5jdl6i2WAcBIOZfZIlk\nepuV5e2YgaCi5gAPvLub2kPNdqd0qaWlxe6EbkmbdSb3mdxm8lGRk7HNMQPB9n2HeGptiG37D9qd\nIoRIkH379lFeXk55eTl5eXkUFhZ2XG9qaurxsStXruT6668/5vc48cQT+ys3Jj/72c/67bkcs7G4\ncEjk03Y7Qw1MLfLZXHM0j8djd0K3pM06k/tMbktJ6Z+/UXNzc1m9ejUAt99+O16vl+9+97sdt/d0\nSIaZM2cyc+bMY7YtX768X1pj9bOf/Yxbb731sGlWl5tj3hEU+iIDQdXHDTaXdM3q8ZMSQdqsM7nP\n5Lb09PS4PfeXv/xlrr76ambPns2NN97IihUrmDNnDtOnT+fEE0/kww8/BOD111/nnHPOASKDyFe/\n+lXmzp3LhAkTuPvuuzuez+v1dtx/7ty5XHjhhZSVlXHZZZd1rKp56aWXKCsrY8aMGVx//fUdz9vZ\n+vXrmTVrFuXl5UydOpVNmzYB8Oijj3ZMX7hwIa2trdx88800NDRQXl7OZZdd1vEcVpdb3N4RKKVG\nAI8Aw4mcqPt+rfXvjrjPXOA5YGt00j+01j+JR092hpsMdwo7Q2Zu6KmpqSEvL8/ujC5Jm3Um9yW6\n7cf/XM+GnQd6dV+tda82ek4sGMxt506KuaWqqorly5eTmprKgQMHWLp0KS6Xi9dee41bb72Vv//9\n70c9pqKigiVLlrB//34mT57MNddcc9QeRO+//z7r16+noKCAk046iWXLljFz5kwWLlzIm2++yahR\no7jkkku6bLrvvvv45je/yWWXXUZTUxOtra1s3LiRp556imXLluF2u/n617/OY489xl133cU999zT\n8W6nndVdW+O5aqgFuEFr/Z5SKgtYpZR6VWu94Yj7LdVaHz089jOlFEMzU6kOHYr3t7IkFAoZ+wtD\n2qwzuc/ktt4OBFZ94QtfIDU1FYDa2lquuOIKNm3ahFKK5uaudyg5++yzSU9Px+fzMWzYMHbv3k1R\nUdFh95k1a1bHtPLycgKBAF6vl9GjR3fs13/JJZdw//33H/X8c+bM4Y477qCqqorPfe5zjB07lkWL\nFrFq1SqOP/54IHJQuWHDhnU7Xy0tLWYNBFrrXcCu6OU6pdRGoBA4ciBImGGZLmPfEQiR7GL5yz3e\nR/jMzMzsuPzDH/6QefPm8eyzzxIIBJg7d26Xj+m82iU1NbXLva56c5/uXHrppcyePZsXX3yRBQsW\n8Mc//hGtNVdccQV33nlnr5/HioRsLFZKlQDTgXe7uHmOUmoNsBP4rtZ6fRePvwq4CqCoqIiKioqO\n24qLi4HDD7bk9/vx+/1UVlZ2vBAej4dhmW7e3nHwsMeXlpYSDoeprq7umJaXl4fP5zvsfl6vl6Ki\nIqqqqqivr++YXlZWRigUoqampmNaYWEhHo+HzZs3d0zz+Xzk5eURCAQ69kN2uVyMGTOGYDBIMBjs\n+H6xzFNJSQk1NTWEQqG4zVMwGKSurs7SPFl5nWKZJyDhr1Ms8xQMBgkEAgl5nWKdJyDur1Nzc3PH\nBtmGhk+2z6WmppKWltaxCqRdRkYGLS0ttLS0dNw/LS0NpRSNjY0d93O5XLjdbhobG2lrawMi7/o9\nHk/H92zX/su5ubmZ5uZmWltbO24Ph8Ps37+foUOH0tjYyEMPPQREBqLGxkZaW1tpa2ujra2N5uZm\nGhoaDnvu9saGhoaOdxLt89TS0kJTUxPjx49ny5YtVFRUUFxczOOPP47Wmra2tsPmafv27YwbN46F\nCxeyZcsWVq1axRlnnMEXv/hFrrvuOnJycti/fz/19fWMHTsWt9vNgQMHOt4BtL+m4XCY5uZmKioq\nDnudeqS1jusX4AVWAZ/r4rbBgDd6eQGw6VjPN2PGDG3V716t0MU3vaAPNbZYfo54aWpqsjuhW9Jm\nncl9iWjbsGGDpce1trb2c4nWt912m/7lL3+pr7jiCv3MM890TF++fLkeO3asLi8v19///vd1cXGx\n1lrrJUuW6LPPPvuwx7a3TZo0SW/dulVrrXVmZuZR99da62uvvVb/5S9/0Vpr/fzzz+vx48fr4447\nTi9cuFBfeumlR/XdeeedeuLEiXratGn6rLPO0vv27dNaa/3kk0/qadOm6SlTpujjjjtOv/3221pr\nrW+88UZdVlZ22HO1L7euljuwUnfze1XpOH44QinlBl4A/q21/k0v7h8AZmqtg93dZ+bMmdrqiWke\nX17Jrc9/yKIbTqN0qNfSc8RLXV0dWVlZdmd0SdqsM7kvEW0bN25kwoQJMT+utbW1Yx2+aay01dfX\n4/V60Vpz7bXXMnbsWL797W/Hra2r5a6UWqW1Pnp/WOK4+6iKbOl5ENjY3SCglMqL3g+l1Kxoz754\nNaU11wFQbeAupJ1XD5hG2qwzuc/ktmN92MtOVtoeeOABysvLmTRpErW1tSxcuDAOZdaXWzy3EZwE\nXA6sVUq17+N0KzASQGt9H3AhcI1SqgVoAC7WcXyLMjQzMrs7Q+YNBEKI5PXtb387Lu8A+ks89xp6\nC+hx/y+t9T3APfFqOJJ/kIsUBdUyEAiRMDrOu4KKw1n5W9oxnywGKCzIJ2+wx8iBwNT9uUHa+sLk\nvkS0eTwe9u3bF/MvJ5MP9Wxym8vlYt++fTEfPsQxxxqCyG50hUMyjFw15POZd/yjdtJmncl9iWhr\n3+117969cf9eIsLj8Rz1QbdjcdRAUFFRQYEvg/e2f2x3ylEqKiooKyuzO6NL0madyX2JaHO73TGd\nKaud05ebVRUVFZaWt6NWDQEU+DKoqQ3T2mbuMcWFECKRHDcQFPoyaG7V7K1rPPadhRDCARw1EHi9\n3o7DUZu2wbj9ULYmkjbrTO6TNmuSsc1RA0FRUVHHCWpMGwhi3biTSNJmncl90mZNMrY5aiCoqqqi\nwPfJmcpMUlVVZXdCt6TNOpP7pM2aZGxz1EBQX1+PN91FdobbuIGg81ElTSNt1pncJ23WJGObowaC\ndgW+DCOPNySEEHZw5EBQ6DPz08VCCGGHuB6GOh76chjqdrc9t45/vF/N2tvP6qcqIYQwmy2HoTZR\n+5mUCnwZ1IVbOBDu+tykduh8lifTSJt1JvdJmzXJ2OaogaD9dG3tu5CatMH4mKeSs5G0WWdyn7RZ\nk4xtjhoI2pm6C6kQQtjBkQNBUceni8M2lwghhP0cNRAUFhYC4Pem405VRu1C2t5mImmzzuQ+abMm\nGdscNRC0n6whJUWRn23WeQliPZFEIkmbdSb3SZs1ydjmqIFg8+bNHZcLfRlGfZagc5tppM06k/uk\nzZpkbHPUQNBZgc+sdwRCCGEXxw4EhUMy2H0gTHNrm90pQghhK0cNBJ3P0Vro89CmoabWjD2HnH5u\nW6tMbgOz+6TNmmRsc9RAkJeX13G50DcIMOezBJ3bTCNt1pncJ23WJGObowaCQCDQcbnAF9m6bsoG\n485tppE260zukzZrkrHNUQNBOPzJaiDTPl3cuc000madyX3SZk0ytjlqIOjM407F700z5h2BEELY\nxVEDgcvlOux6gS/DmMNMHNlmEmmzzuQ+abMmGdsceT6Cdtc8uoqPdtex6Ia5/fJ8QghhKjkfQVQw\nGDzseuRDZWFMGAyPbDOJtFlncp+0WZOMbY4fCBqaW/n4kP0nqEnG/1yJYHIbmN0nbdYkY5ujBoIj\nFRq255AQQthBBgLM+SyBEELYwVEDQXFx8WHX209ZacJ5CY5sM4m0WWdyn7RZk4xtjhoIjjRkkBuP\nO0VWDQkhHM1RA8G2bdsOu66UMua8BEe2mUTarDO5T9qsScY2Rw0EXZHzEgghnM7xA4Ep7wiEEMIu\njhoI/H7/UdMKfRkE65sIN7faUPSJrtpMIW3WmdwnbdYkY1vcBgKl1Ail1BKl1Aal1Hql1De7uI9S\nSt2tlKpUSn2glDouXj3Q9UJqPwrpLptPUJOM/7kSweQ2MLtP2qxJxrZ4viNoAW7QWk8ETgCuVUpN\nPOI+nwHGRr+uAv43jj1UVlYeNc2UXUi7ajOFtFlncp+0WZOMbXEbCLTWu7TW70Uv1wEbgcIj7nY+\n8IiOeAfwKaXy49XU0tJy1DRTPl3cVZsppM06k/ukzZpkbEvI8VSVUiXAdODdI24qBHZ0ul4Vnbbr\niMdfReQdA0VFRVRUVHTc1v4Bis67Tfn9fvx+P5WVlR0LxuOJnJGspqaGUCjUcd+RJaNQwJrKHUzN\nOghETvfm8/kO+z5er5eioiKqqqqor6/vmF5WVkYoFKKmpuaTmSosxOPxsHnz5o5pPp+PvLw8AoFA\nx8kjXC4XY8aMIRgMEgwGO75fLPNUUlJy1DyVlpYSDoeprq7umNaXeQoGg9TV1VmaJyuvUyzzBCT8\ndYplnoLBIIFAICGvU6zzBCTsdYp1njr/PCTidYplnoLBIKFQyJbfEceaJ6DbeeqR1jquX4AXWAV8\nrovbXgBO7nR9ETCzp+ebMWOGtmrr1q1dTp99x2v6hqdXW37e/tBdmwmkzTqT+6TNmoHaBqzU3fxe\njeteQ0opN/B34DGt9T+6uEs1MKLT9aLotLgoKSnpcnqBz2P7NoLu2kwgbdaZ3Cdt1iRjWzz3GlLA\ng8BGrfVvurnb88CXonsPnQDUaq13dXPfPuvu7VHhkEHsrLV3IDjmWzcbSZt1JvdJmzXJ2BbPdwQn\nAZcD85VSq6NfC5RSVyulro7e5yVgC1AJPAB8PY49h63366zA52FXKExbm30nqOmuzQTSZp3JfdJm\nTTK2xW1jsdb6LUAd4z4auDZeDb1V5MugqbWNYH0jwwZ77M4RQoiEctQni7tTIOclEEI4mKMGgtLS\n0i6nd3yozMaBoLs2E0ibdSb3SZs1ydjmqIGgfd/cIxUY8KGy7tpMIG3WmdwnbdYkY5ujBoLOH3Lp\nbLDHTVa6y9ZdSLtrM4G0WWdyn7RZk4xtjhoIelI4JIPqkLkjvRBCxIsMBFFyghohhFM5aiBoPzZN\nVwp9GVR9fKj9UBcJ11Ob3aTNOpP7pM2aZGxz1EDg8/m6va18hI8D4RZW77DnwyI9tdlN2qwzuU/a\nrEnGNkcNBJ2PFHikMyYOJy01hRc/iNsRLnrUU5vdpM06k/ukzZpkbHPUQNCT7Aw3p47z89LaXbYe\nakIIIRJNBoJOzp6az87aMO/btHpICCHs4KiBwOv19nj7GROGk+ayZ/XQsdrsJG3WmdwnbdYkY5uy\nay8Zq2bOnKlXrlwZt+f/70dWsraqluU3zyclpcdj5gkhxIChlFqltZ7Z1W2OekdQVVV1zPucMzWf\nmgNh3t/xcQKKPtGbNrtIm3Um90mbNcnY5qiBoPN5RLtzenT10AsJXj3Umza7SJt1JvdJmzXJ2Oao\ngaA3vOku5o4bKnsPCSEcQwaCLpw9NZ/dBxpZtT2xq4eEEMIOsrG4C/WNLcz4f69yyayR3H7epLh+\nLyGESATZWBzV2/N5etNdzBs/jJfW7qI1QauHkvE8qIlgchuY3Sdt1iRjm6MGgpqaml7f9+yp+eyp\na2RlYH8ciz4RS1uiSZt1JvdJmzXJ2OaogSAW88uG4XGn8OJae449JIQQiSIDQTcy013MLxvGS2tr\nErZ6SAgh7OCogaCwsDCm+589pYBgfSMrtsZ/9VCsbYkkbdaZ3Cdt1iRjm6MGAo/HE9P955UNJcOd\nyotrd8ap6BOxtiWStFlncp+0WZOMbY4aCDZv3hzT/QeluZg/YRgvr4v/6qFY2xJJ2qwzuU/arEnG\nNkcNBFacMyWfYH0T727dZ3eKEELEhQwExzB3/DAGpaXaduYyIYSIN0cNBFbO55mRlsr8ssjqoZbW\ntjhURSTjeVATweQ2MLtP2qxJxjZHDQR5eXmWHnfO1Hz2HWzi3TjuPWS1LRGkzTqT+6TNmmRsc9RA\nEAgELD2uffXQP9fEb+8hq22JIG3WmdwnbdYkY5ujBoJwOGzpcR53KudNK+Dv71Xx0e66fq6KsNqW\nCNJmncl90mZNMrY5aiDoi++dNR5vuotb/rFWzlMghEgqvRoIlFKZSqmU6OVxSqnzlFLu+Kb1P5fL\nZfmxud50fnD2RFZt+5jH3t3Wj1URfWmLN2mzzuQ+abMmGdt6dT4CpdQq4BRgCLAM+A/QpLW+zNJ3\n7YNEnI+gO1prvvTnFby/PcSr3zmV/OwMWzqEECJW/XE+AqW1PgR8DviD1voLwIA7Y0swGOzT45VS\n3PHZKbS0tXHbc+v7qSqir23xJG3WmdwnbdYkY1uvBwKl1BzgMuDF6LRUS9/RRv3xAo7MHcS3zhjH\nKxt28/K6/vuQWTL+50oEk9vA7D5psyYZ23o7EHwLuAV4Vmu9Xik1Glhi6Tsmga+dPIqJ+YP50XPr\nqW1otjtHCCH6pFcDgdb6Da31eVrrn0c3Gge11tfHuc1YrtQUfv75qQTrG/n5yxV25wghRJ/0dq+h\nx5VSg5VSmcA6YINS6nvHeMyflVJ7lFLrurl9rlKqVim1Ovr1o9jzY1NcXNxvzzWlKJuvnjSKx9/d\n3i/nK+jPtv4mbdaZ3Cdt1iRjW29XDU3UWh8APgv8CxgFXH6MxzwEfPoY91mqtS6Pfv2kly3G+M6n\nxlE0JINb/vEBjS2tducIIYQlvR0I3NHPDXwWeF5r3Qz0uN+p1vpNIDFnfu+lbdv6d///QWku7rhg\nCpv3HuTeJX07Rnl/t/UnabPO5D5psyYZ23r76YM/AgFgDfCmUqoYOGDpOx5ujlJqDbAT+K7Wust9\nMpVSVwFXARQVFVFR8cl6+fa3Qp0XgN/vx+/3U1lZSUtLC/DJmXtqamoIhUId9y0tLSUcDlNdXd0x\nLS8vD5/Pd9j38Xq9FBUVUVVVRX19fcf008rKWDBpKH9YsomJ3jAlQ9IoLCzE4/EcdpIIn89HXl4e\ngUCg42PgLpeLMWPGEAwGCQaDHd8vlnkqKSnp93kqKysjFApRU1MDRPZEqKurszRPVl6nWOYJsDRP\ngOXXKZZ5CgaDBAKBhLxOsc4TkLDXKdZ56vzzkIjXKZZ5CgaDhEKhhL1OscwT0O089UhrbekLcPXi\nPiXAum5uGwx4o5cXAJt6831nzJihrdq4caPlx/YkWBfW03/yij7xzkV66956S88Rr7b+IG3Wmdwn\nbdYM1DZgpe7m92pvNxZnK6V+o5RaGf36NZDZm8f2MAAd0FrXRy+/RGT1k78vz3ks7SNmf8v1pvPI\nV2dxqKmFL/zxbT6sif3AdPFq6w/SZp3JfdJmTTK29XYbwZ+BOuCL0a8DwF8sfccopVSeUkpFL8+K\ntsT1fJDxfAEnF2bz9MI5KOCi+9/mg6rQMR/TWTL+50oEk9vA7D5psyYZ23o7EJRqrW/TWm+Jfv0Y\nGN3TA5RSTwBvA+OVUlVKqSuVUlcrpa6O3uVCYF10G8HdwMXRty9xU1lZGc+nZ+zwLJ65eg6ZaS4u\nfeBd/hPo/bbyeLf1hbRZZ3KftFmTjG29HQgalFInt19RSp0ENPT0AK31JVrrfK21W2tdpLV+UGt9\nn9b6vujt92itJ2mtp2mtT9BaL7c0BzFo34AST8W5mfztmjkMG5zO5Q++y9JNe3v1uES0WSVt1pnc\nJ23WJGNbbweCq4F7lVIBpVQAuAdYaOk7OkB+dgZPL5xDSW4mVz60kn+vP8YWeyGEsFFvDzGxRms9\nDZgKTNVaTwfmx7UsDtp3IU0EvzedJ686gYkFg/n6Y+/xf+9X93j/RLbFStqsM7lP2qxJxrZenY+g\nywcqtV1rPdLSg/vAzvMRWFHf2MLXHv4P727dzzfmjeG6+WNJc8mJ4YQQidUf5yPo8nn78FhbHPND\nFXHgTXfx0FdmcUF5IXcvruSz9y5j466jP4tnR1tvSZt1JvdJmzXJ2NaXgWDAnbi386cFE8njTuU3\nF5Xzx8tnsKcuzHn3vMU9izfR0tpme1tvSJt1JvdJmzXJ2NbjISaUUnV0/QtfAXKexhidNSmP40ty\n+OFz6/jVKx/x6obd/PqL0xgzLMvuNCGEg/X4jkBrnaW1HtzFV5bW2twzOBssJzONey89jnsunc72\n/YdYcPdb3P/mZlrbBtwbLCFEkrC8sdgufdlY3NzcjNvt7uci6/bWNfL9Z9fyyobdHDfSx4/Pm8yU\nomy7s45i2nLrzOQ2MLtP2qwZqG3x2lg84LQf0c8UQ7PS+ePlM/jtReVs2VvPufe8xfVPvM+O/Yfs\nTjuMacutM5PbwOw+abMmGdscNRB0PjSuKZRSfHZ6IQ9eUMS180p5ZUMN83/9Oj/+53r2H2yyOw8w\nc7m1M7kNzO6TNmuSsc1RA4HJMtNS+d5ZZbz+3Xl8/rgiHl4e4LRfLOHeJZU0NMnZz4QQ8SMDgWHy\nsj3c9fmp/PtbpzJ7dC6//PeHzP3VEp5YsZ2mlrZjP4EQQsTIUQNB+xmtTHRk29jhWfzpipk8vXAO\nBb4MbvnHWub+cgkPLdtKuDmx7xAG0nIzjcl90mZNMrY5aq+hgUprzRsf7eXeJZX8J/Axfm8aXz15\nFJefUEyWx8y9F4QQZpG9hqI6n1/UND21KaWYO34Yz1x9Ik9ddQITC7L5xcsfcuJdi/n1Kx/GfaPy\nQF1uJjC5T9qsScY2+VDYADN7dC6zR+fyQVWIPyzZzO8XV/KnpVu5ZNZIvnJSCSNyBtmdKIQYYGQg\nGKCmFvm47/IZbNpdxx9e38wjbwd4aPlWPjM5n6+dMorpI4fYnSiEGCAcNRB4vV67E7pltW3s8Cz+\n56JyvnfWeB5eHuDxFdt5ce0uZhYP4WunjOLMiXmkpvTtQLHJuNwSxeQ+abMmGdtkY3GSqW9s4en/\n7ODPy7ZS9XEDI3MG8dWTSvjCzBFkpjtq3BdCdCIbi6OqqqrsTuhWf7V501189eRRvP7dufzhsuPI\n9aZx+z83cMKdi/jpCxvYvi/2w1c4YbnFi8l90mZNMrY56k/E+vp6uxO61d9trtQUFkzJZ8GUfFZt\n+5iHlgd4aHmAB5dt5fSy4XzlpBJOLM1FqWOvNnLScutvJvdJmzXJ2OaogcCpZhQPYUbxEGoWTOCx\nd7fx+LvbeW3jbsYN93LFiSVcML2QQWnyX0EIp3LUqiGny8v2cMOnxrPs5vn88sKpuFNT+P6z6zjh\nZ4u4818b2VXbYHeiEMIGsrHYwbTWrNz2MX9ZtpWX19WQohTnTivgypNHMbnQvPMiCCGsk43FUcl4\nrtG+UEpxfEkOf7hsBm98bx5fmlPCK+trOOf3b3Hx/W+zaONu2tq0LLc+MLlP2qxJxjZHDQQ1NTV2\nJ3TL7rYROYP40bkTWX7L6dy6oIxt+w5x5cMrOeN/3uDBNz5M+IHuesvu5XYsJvdJmzXJ2OaogUAc\nW3aGm6tOLeXNG+fxu4vLyUxz8fu3g5x012J+v2gToUNmnCxHCNF/ZFcR0SV3agrnlxdy3rQCnnlj\nNS8HWvj1qx/xh9c3c9HxI7jy5FFyXCMhkoSjNhbX1dWRlZXVz0X9YyC0fVhTx/1vbuG51dVo4Jyp\n+Vx16mgmFdi3Ydnk5QZm90mbNQO1raeNxY4aCJqbm3G7zTx+/0Bq21XbwF+WBXj83e3UN7Zwylg/\n15xWypxefkAtnm2mMblP2qwZqG2y11DU5s2b7U7o1kBqy8/O4NYFE1h283xu+nQZFTV1XPqnd/ns\nvct4ed0uWtsS98eFycsNzO6TNmuSsc1RA4HoX9kZbq6ZW8rSG+fxswumEGpo5upH3+PM37zBU//Z\nTmOLmXsaCSEOJwOB6DOPO5VLZ49k8Q1zuefS6WSkpXLT39dy6i+W8MCbW6hvbLE7UQjRA0cNBD6f\nz+6EbiVDW2qK4pypBbzwjathMTsAABeTSURBVJP565WzKB3q5Y6XNnLSXYv5n1c/isuupyYvNzC7\nT9qsScY2R20sFom3ekeIe5dU8uqG3WSmpfJfJxRz5SmjGJblsTtNCEeRjcVRgUDA7oRuJWtb+Qgf\nD3xpJi9/6xROnzCcB5Zu4eSfL+FHz62jOtT3g9yZvNzA7D5psyYZ2xw1EITDYbsTupXsbWV5g7n7\nkuksvmEun5teyBMrtnPaL5bwvWfWsDV40Na2eDK5T9qsScY2Rw0Ewn4l/kzu+vxU3vjePP7rhGL+\n+cFOTv/163zn6dUE+jAgCCGsi9tAoJT6s1Jqj1JqXTe3K6XU3UqpSqXUB0qp4+LV0s7lMveIGk5r\nK/BlcPt5k1h643yuPHkUL63dxem/eYPvPrOGbft6PyCYvNzA7D5psyYZ2+K2sVgpdSpQDzyitZ7c\nxe0LgG8AC4DZwO+01rOP9byysTg57akL88c3tvDoO9toadN8bnoh35g/lpG5cjwjIfqDLRuLtdZv\nAvt7uMv5RAYJrbV+B/AppfLj1QMQDAbj+fR94vS2YVkefnjORJbeOI8vzSnmuTU7mf/r17npbx+w\nY/8hW9v6wuQ+abMmGdvs3EZQCOzodL0qOi1ukvEFTIREtg0b7OG2cyex9MbINoRnV1cz71evc/Pf\nux4QTF5uYHaftFmTjG3mruzqRCl1FXAVQFFRERUVFR23FRcXA7Bt27aOaX6/H7/fT2VlJS0tkU+1\nejyR/dZramoOO4tPaWkp4XCY6urqjml5eXn4fL7Dvo/X66WoqIiqqirq6+s7ppeVlREKhQ47IURh\nYSEej+ew4374fD7y8vIIBAIdW/ZdLhdjxowhGAwSDAY7vl8s81RSUhL3eQoGg9TV1VmaJyuvU0lJ\nCfpQiIvHpXJG4QieXhfiH+9X87dVVZxR6uXiqUPIy3KTl5cHkPDXKZZ5CgaDBAKBhLxOsc4T0OfX\nKV7z1PnnIRGvUyzzFAwGCYVCtvyOONY8Ad3OU0/i+oEypVQJ8EI32wj+CLyutX4iev1DYK7WeldP\nz9mXbQQVFRWUlZVZemy8SVvPamrD3PfGZh5fsZ22Ns2FM4q4dt4YDu7ZbntbT0xYdt2RNmsGaptt\nh6E+xkBwNnAdn2wsvltrPetYz9mXgaChoYGMjAxLj403aeudIweE86fl8c0zyozdqGzSsjuStFkz\nUNt6GgjitmpIKfUEMBfwK6WqgNsAN4DW+j7gJSKDQCVwCPhKvFpE8sjL9nD7eZO4+rTSyIDw7nb+\nb00NF0wv5Np5Yxjlz7Q7UYgBJ557DV2itc7XWru11kVa6we11vdFBwGiewtdq7Uu1VpP0VrHfZ/Q\nzuvTTCNtsWkfEP7y+RFcMaeEf66JfjDtqdVs3lt/7CdIEBOXXTtpsyYZ2+STxWJAyx3k4kfnTmTp\nTfP42imj+de6Gs74zRtc/8T7fLS7zu48IQYEGQhEUhiW5eHWBRN466Z5XH1aKYs27uas377J1x9b\nxfqdtXbnCWG0AbH7aH9p373KRNJmzZFtud50bvp0GVedMpoH39rKw8sDvLS2hjMmDOPaeWOYPnKI\nrX0mkTZrkrFNzkcgklptQzOPLA/w4LKthA41c8pYP9fNG8Ps0bl2pwmRUHI+gqjKykq7E7olbdYc\nqy07w803Th/Lspvmc+uCMjbuquOi+9/hC/ct542P9hLvP4QG8rKzk7RZY7XNUQNB+6ftTCRt1vS2\nLTPdxVWnlvLWTfP48XmTqPq4gSv+vILz713Gv9fX0NYWnwEhGZadHaTNGqttjhoIhPC4U7nixBLe\n+N487vzcFEKHmln411V8+ndv8tzqalpa2+xOFCLhHDUQtB9vyETSZo3VtjRXCpfMGsniG07jtxeV\nozV888nVnP6bN3hyxXaaWvpnQEjGZZcI0maN1TbZWCwE0NameWXDbu5dUsna6lrysz1cdepoLj5+\nJBlpqXbnCdFnsrE46lhH4LOTtFnTX20pKYpPT87j+etO4uGvzmLEkEH8+J8bOPnni7l3SSUHws22\n9sWDtFmTjG2OGgg6H1rWNNJmTX+3KaU4bdxQnr56Dk8vnMPkwmx++e8POemuxfzq3x+y/2CTrX39\nSdqsScY2R32gTIhYzBqVw6xRs1hXXcu9Syq59/VKHnxrK5fMGsl/nzqK/Gwzj0ApRKxkIBDiGCYX\nZvO//zWDyj11/O/rW3j47QB/fSfAhTOK+MpJoxg3PMvuRCH6xFEbi5ubm3G73f1c1D+kzRo72nbs\nP8T9b27hqZU7aGpp47iRPi6eNZJzpuYzKO3wv61k2Vkjbdb01CYbi6PaT/9mImmzxo62ETmD+H+f\nncw7t5zOD86eQG1DMzf+7QNm37GI7z+7lnXVnxzkTpadNdJmjdU2Rw0Enc+jahpps8bOtpzMNL52\nymhe+85pPHP1HM6cNJy/rarinN+/xTm/X8qj72xjU2CHbX3HIq+rNcnYJtsIhOgjpRTHl+RwfEkO\nt507iedWV/PEih384P/WMcituHBLG5efUMxY2ZYgDCUDgRD9KDvDzZfmlHD5CcWsqarlnpfX8OSK\nHTzy9jbmjM7lS3OKOXPicFypjnozLgznqIEgLy/P7oRuSZs1prYppSgf4eNXX5hGqyuDp1bu4LF3\ntnPNY++RN9jDpbNHcvGsEQzLsu9wBaYuO5A2q6y2OWqvISHs1NqmWVyxh0feDrB0U5DUFMUpY/2c\nX17AmRPz8KY76u8ykWCy11BURUWF3QndkjZrTG6Dw/tSUxRnThzOX6+czeIbTuO/TxnNpt31fPup\nNcz86atc9/h7vLphd78d8C6WNtNImzVW2+RPECFsMHqol5s/U8aNZ43nve0f89zqnby4dhcvfLCL\n7Aw3C6bkce60AmaPyiU1RdmdK5KcDARC2CglRTGzJIeZJTn86NyJvFUZ5PnVO3lu9U6eWLEDvzed\nBVPyOHtKPseX5JAig4KIA0cNBF6v1+6EbkmbNSa3QWx97tQU5o0fxrzxw2hoamVxxR5eXLuTp1dG\n9joaPjidz0zO59xp+UwfMaTPg4LJy07arLHaJhuLhTDcwcYWFlXs4YU1O3n9o700tbRRkO3hU5Py\nOGPCcGaNyiHN5ajNfcKCnjYWO2ogqKqqoqioqJ+L+oe0WWNyG/R/X124mdc27uaFNbtYWhmkqaWN\nrHQXp44byukTIu8mhmSm2dLWn6TNmp7aehoIHLVqqL6+3u6EbkmbNSa3Qf/3ZXncXDC9iAumF3Go\nqYVllftYtHE3iyr28OLaXaQomFE8hNMnDOeMCcMoHepFqa5XIZm87KTNGqttjhoIhEgmg9JcnDlx\nOGdOHE5bm2ZtdS2LNu7mtY17uOtfFdz1rwpG5gxiftkwTp8wjFmjckh3yWk3xdFkIBAiCaSkKKaN\n8DFthI/vfGo8O0MNLK7Yw+KKPTyxYjsPLQ+QmZbKqeOGMr9sGPPKhtmdLAziqG0EQjhRQ1MryzcH\neW3jHhZX7Gb3gUYAyvKyOGF0LrNH5TBrVA653nSbS0U8ycbiqFAohM/n6+ei/iFt1pjcBub1aa1Z\nv/MAr3+4h7c+2sOa6joamlsBGDfcGx0Ycpk1KoehWfYNDKYtt84GaptsLI6qqakx9gWUNmtMbgPz\n+pRSTC7MZnJhNmcUtDJ6zBzWVod4Z8t+3t26n7+tquKRt7cBMDJnENNH+pg+wsf0kUOYkD84Ybup\nmrbcOkvGNkcNBEKIw6W5UphRnMOM4hyunQfNrW2sq65lxdb9rN4R4p0t+3hu9c6O+04pzGZ6dFvE\n5MJsinMGyaedk4AMBEKIDu7UFKaPHML0kUM6pu2qbeD97SHe3/4x728P8dd3tvGnt7YCkJXuYmLB\n4Oi7jMFMLshm9FCvHB9pgHHUQFBYWGh3QrekzRqT28Dsvt625WdnkD8lgwVT8gFoamnjo911rN9Z\ny7rqA6zbWcuj72yjMXrU1Ax3KhMLBjO1KJupRdlMKfQx2p8Z0zuHZFhudrDa5qiNxc3Nzbjd7n4u\n6h/SZo3JbWB2X3+2tbS2sXnvQdZV17K2upZ11bWs33mgY0O0N93F5MLBTC3yMaUwm7HDvZTkZuJx\nd/25Bqcst/7WU5tsLI7avHkzZWVldmd0SdqsMbkNzO7rzzZXagrj87IYn5fF52dEDnHQPjh8UBVi\nbXUta6pqeWh54LDzLeRneyjJzaTEn8ko/yCKczMZ7c+kIVjF1EkT+qWtvyXja+qogUAIkTidB4cv\nzBwBRFYrbdpTx+a9BwkEI19b9x3k5XW7+PhQc8djUxSMzKmhdKiX0mFeSodmMnqol9KhXnJ6eSwl\n0XsyEAghEibNlcKkgmwmFWQfdVvtoWYC+w6yNXiQFRXbqG3zsHlvfcfB9doNGeSOvIPIzWSUv/3d\nRORfOd2nNXFdakqpTwO/A1KBP2mt7zri9i8DvwSqo5Pu0Vr/KV49pu77C9JmlcltYHafaW3Zg9xM\nGxTZNfWE/NSOE7G3tmmqP25gc7CezXvqO95NvL1lH/94v/qw5xialU5xziAKfBnk+zwUZGeQn+2h\nwJdBgS+DIYPc3R6Er7dMW26dWW2L28ZipVQq8BFwJlAF/Ae4RGu9odN9vgzM1Fpf19vnlUNMCCHa\nNTS1EtgXGRi2RFc1bd9/iF21YWpqwzS1Hn7+Z487hQJfBkVDBjFiSOTfoiEZjMiJ/JubmdbngcJU\ndm0sngVUaq23RCOeBM4HNvT4qDgKBAKUlJTY9e17JG3WmNwGZvclQ1tGWioT8gczIX/wUbe1tWmC\nBxvZFQqzq7aBndF/q0MN7NjfwNqq0GHbJSCy6+vQrHRyvWnkZqaRk5lGTmY6fm/kcq43naYDQaaM\nHUVOZppxJwSy+prGcyAoBHZ0ul4FzO7ifp9XSp1K5N3Dt7XWO468g1LqKuAqgKKiIioqKjpuKy4u\nBmDbtm0d0/x+P36/n8rKSlpaWgDweDyEw2FqamoIhUId9y0tLSUcDlNd/clbzLy8PHw+32Hfx+v1\nUlRURFVV1WHH/C4rKyMUClFTU/PJjBcW4vF42Lx5c8c0n89HXl4egUCAcDgMgMvlYsyYMQSDQaqq\nqjqmxzJPJSUlcZ+nYDBIbm5uzPMUDAYtvU6xzFM4HE7o6xTrPLXfPxGvU6zz1N6XiNcp1nnq/PPQ\n19dpXHEx4/zpkXnypwCZ+P3F+P1+PtjwEdW1Deyub2F/GA6qDHbsrWXvgQa21Bzi/cZWasNttLQd\nueZkS2Q+0lLweVLxZaTi96YzYqiPtLYwmaltDMlIZUhGKjMmjcPd2kBo315Lr1Ms//fC4XC3r1NP\n4rlq6ELg01rrr0WvXw7M7rwaSCmVC9RrrRuVUguBi7TW83t63r6sGqqoqDB2ty9ps8bkNjC7T9p6\nR2vNgXAL+w82sa++kTUfbsHjG8q++sj14MEmgnWN7DvYRLC+kdAR7zLaZaW7yPWmMSQzjSGD2r/c\nHddzMt0MznAz2OPGm+4iy+Miy+OO6V1HT8vNrlVD1cCITteL+GSjMABa632drv4J+EUce3C5zN2j\nQNqsMbkNzO6Ttt5RSpGd4SY7w80ofya+lv2MGVPc7f2bWtrYd7CRYF0Te+vD0X8b2VvX2DFQ7D4Q\npmLXAT4+1NzxobvupLlSGOxx4U138V8nFPO1U0Z3e1+ryy2e7whcRFb3nE5kAPgPcKnWen2n++Rr\nrXdFL18A3KS1PqGn55WNxUKIZBJubuXjQ03sP9jEgYYW6sLN1De2UBeOXK6LXq4Pt3D6hGGcX27x\nMBJ2vCPQWrcopa4D/k1k99E/a63XK6V+AqzUWj8PXK+UOg9oAfYDX45XD0TWdfv9/nh+C8ukzRqT\n28DsPmmzpr/bPO7UyPGcsjP6/FxW2+K6yVtr/ZLWepzWulRrfUd02o+igwBa61u01pO01tO01vO0\n1hU9P2PfdN7YYhpps8bkNjC7T9qsScY2s/Z9EkIIkXAyEAghhMM5aiBo3/fWRNJmjcltYHaftFmT\njG2OGgiEEEIczVEDQedP4ZlG2qwxuQ3M7pM2a5KxzVEDgRBCiKPJQCCEEA434M5ZrJTaC1h9b+YH\nTN0JWNqsMbkNzO6TNmsGalux1npoVzcMuIGgL5RSK7v7iLXdpM0ak9vA7D5psyYZ22TVkBBCOJwM\nBEII4XBOGwjutzugB9JmjcltYHaftFmTdG2O2kYghBDiaE57RyCEEOIIMhAIIYTDOWYgUEp9Win1\noVKqUil1s909nSmlAkqptUqp1UopW0+/ppT6s1Jqj1JqXadpOUqpV5VSm6L/DjGo7XalVHV02a1W\nSi2wqW2EUmqJUmqDUmq9Uuqb0em2L7se2mxfdkopj1JqhVJqTbTtx9Hpo5RS70Z/Xp9SSqUZ1PaQ\nUmprp+VWnui2To2pSqn3lVIvRK9bW25a66T/InKGtM3AaCANWANMtLurU18A8NvdEW05FTgOWNdp\n2i+Am6OXbwZ+blDb7cB3DVhu+cBx0ctZRE7TOtGEZddDm+3LDlCAN3rZDbwLnAA8DVwcnX4fcI1B\nbQ8BF9r9fy7a9R3gceCF6HVLy80p7whmAZVa6y1a6ybgSeB8m5uMpLV+k8hpQzs7H3g4evlh4LMJ\njYrqps0IWutdWuv3opfrgI1AIQYsux7abKcj6qNX3dEvDcwH/hadbtdy667NCEqpIuBs4E/R6wqL\ny80pA0EhsKPT9SoM+UGI0sArSqlVSqmr7I7pwnCt9a7o5RpguJ0xXbhOKfVBdNWRLautOlNKlQDT\nifwFadSyO6INDFh20dUbq4E9wKtE3r2HtNYt0bvY9vN6ZJvWun253RFdbv+jlEq3ow34LXAj0Ba9\nnovF5eaUgcB0J2utjwM+A1yrlDrV7qDu6Mh7TmP+KgL+FygFyoFdwK/tjFFKeYG/A9/SWh/ofJvd\ny66LNiOWnda6VWtdDhQRefdeZkdHV45sU0pNBm4h0ng8kAPclOgupdQ5wB6t9ar+eD6nDATVwIhO\n14ui04ygta6O/rsHeJbID4NJdiul8gGi/+6xuaeD1np39Ie1DXgAG5edUspN5BftY1rrf0QnG7Hs\numozadlFe0LAEmAO4FNKuaI32f7z2qnt09FVbVpr3Qj8BXuW20nAeUqpAJFV3fOB32FxuTllIPgP\nMDa6RT0NuBh43uYmAJRSmUqprPbLwKeAdT0/KuGeB66IXr4CeM7GlsO0/5KNugCbll10/eyDwEat\n9W863WT7suuuzYRlp5QaqpTyRS9nAGcS2YaxBLgweje7lltXbRWdBnZFZB18wpeb1voWrXWR1rqE\nyO+zxVrry7C63Oze6p2oL2ABkb0lNgPft7unU9doInsxrQHW290GPEFkNUEzkXWMVxJZ97gI2AS8\nBuQY1PZXYC3wAZFfuvk2tZ1MZLXPB8Dq6NcCE5ZdD222LztgKvB+tGEd8KPo9NHACqASeAZIN6ht\ncXS5rQMeJbpnkV1fwFw+2WvI0nKTQ0wIIYTDOWXVkBBCiG7IQCCEEA4nA4EQQjicDARCCOFwMhAI\nIYTDyUAgxBGUUq2djiy5WvXj0WqVUiWdj54qhAlcx76LEI7ToCOHFRDCEeQdgRC9pCLnjfiFipw7\nYoVSakx0eolSanH0IGSLlFIjo9OHK6WejR7Pfo1S6sToU6UqpR6IHuP+leinVoWwjQwEQhwt44hV\nQxd1uq1Waz0FuIfI0R8Bfg88rLWeCjwG3B2dfjfwhtZ6GpHzKKyPTh8L3Ku1ngSEgM/HeX6E6JF8\nsliIIyil6rXW3i6mB4D5Wust0YO41Witc5VSQSKHZ2iOTt+ltfYrpfYCRTpycLL25yghcjjjsdHr\nNwFurfVP4z9nQnRN3hEIERvdzeVYNHa63IpsqxM2k4FAiNhc1Onft6OXlxM5AiTAZcDS6OVFwDXQ\ncYKT7ERFChEL+UtEiKNlRM9K1e5lrXX7LqRDlFIfEPmr/pLotG8Af1FKfQ/YC3wlOv2bwP1KqSuJ\n/OV/DZGjpwphFNlGIEQvRbcRzNRaB+1uEaI/yaohIYRwOHlHIIQQDifvCIQQwuFkIBBCCIeTgUAI\nIRxOBgIhhHA4GQiEEMLh/j/M747rOPVgTgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "render_training_history(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "X-dhNP2OG2EM"
   },
   "source": [
    "## Generate text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SU_YfP6sG3NC"
   },
   "source": [
    "### Restore the latest checkpoint\n",
    "\n",
    "To keep this prediction step simple, use a batch size of 1.\n",
    "\n",
    "Because of the way the RNN state is passed from timestep to timestep, the model only accepts a fixed batch size once built.\n",
    "\n",
    "To run the model with a different `batch_size`, we need to rebuild the model and restore the weights from the checkpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1197040,
     "status": "ok",
     "timestamp": 1584521003726,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "BlG8o3wiG6f2",
    "outputId": "031b45e0-fcdf-4ab6-b7e9-e167b29800c9"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'tmp/checkpoints/ckpt_40'"
      ]
     },
     "execution_count": 44,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf.train.latest_checkpoint(checkpoint_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "l7evN0LvH01P"
   },
   "outputs": [],
   "source": [
    "simplified_batch_size = 1\n",
    "\n",
    "model = build_model(vocab_size, embedding_dim, rnn_units, batch_size=1)\n",
    "\n",
    "model.load_weights(tf.train.latest_checkpoint(checkpoint_dir))\n",
    "\n",
    "model.build(tf.TensorShape([simplified_batch_size, None]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 255
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1196901,
     "status": "ok",
     "timestamp": 1584521004053,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Y3eduDtZI9zQ",
    "outputId": "95ccdb2f-f059-4be2-ee85-3541f422c7ea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (1, None, 256)            16640     \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (1, None, 1024)           5246976   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (1, None, 65)             66625     \n",
      "=================================================================\n",
      "Total params: 5,330,241\n",
      "Trainable params: 5,330,241\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "euNvAtr4JC3A"
   },
   "source": [
    "### The prediction loop\n",
    "\n",
    "The following code block generates the text:\n",
    "\n",
    "- It Starts by choosing a start string, initializing the RNN state and setting the number of characters to generate.\n",
    "\n",
    "- Get the prediction distribution of the next character using the start string and the RNN state.\n",
    "\n",
    "- Then, use a categorical distribution to calculate the index of the predicted character. Use this predicted character as our next input to the model.\n",
    "\n",
    "- The RNN state returned by the model is fed back into the model so that it now has more context, instead than only one character. After predicting the next character, the modified RNN states are again fed back into the model, which is how it learns as it gets more context from the previously predicted characters.\n",
    "\n",
    "![Prediction loop](https://www.tensorflow.org/tutorials/text/images/text_generation_sampling.png)\n",
    "\n",
    "Image source: [Text generation with an RNN](https://www.tensorflow.org/tutorials/text/text_generation) notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bOqdqGouJFf_"
   },
   "outputs": [],
   "source": [
    "# num_generate\n",
    "# - number of characters to generate.\n",
    "#\n",
    "# temperature\n",
    "# - Low temperatures results in more predictable text.\n",
    "# - Higher temperatures results in more surprising text.\n",
    "# - Experiment to find the best setting.\n",
    "def generate_text(model, start_string, num_generate = 1000, temperature=1.0):\n",
    "    # Evaluation step (generating text using the learned model)\n",
    "\n",
    "    # Converting our start string to numbers (vectorizing).\n",
    "    input_indices = [char2index[s] for s in start_string]\n",
    "    input_indices = tf.expand_dims(input_indices, 0)\n",
    "\n",
    "    # Empty string to store our results.\n",
    "    text_generated = []\n",
    "\n",
    "    # Here batch size == 1.\n",
    "    model.reset_states()\n",
    "    for char_index in range(num_generate):\n",
    "        predictions = model(input_indices)\n",
    "        # remove the batch dimension\n",
    "        predictions = tf.squeeze(predictions, 0)\n",
    "\n",
    "        # Using a categorical distribution to predict the character returned by the model.\n",
    "        predictions = predictions / temperature\n",
    "        predicted_id = tf.random.categorical(\n",
    "        predictions,\n",
    "        num_samples=1\n",
    "        )[-1,0].numpy()\n",
    "\n",
    "        # We pass the predicted character as the next input to the model\n",
    "        # along with the previous hidden state.\n",
    "        input_indices = tf.expand_dims([predicted_id], 0)\n",
    "\n",
    "        text_generated.append(index2char[predicted_id])\n",
    "\n",
    "    return (start_string + ''.join(text_generated))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 578
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1202383,
     "status": "ok",
     "timestamp": 1584521010077,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "Z-8e8P60J5Pg",
    "outputId": "25240489-3063-4bdd-b203-ef14886d1e60"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO: God do not burden me.\n",
      "\n",
      "GREMIO:\n",
      "I want the king that murders writ upon your tongue\n",
      "And born too supplict, but thanks\n",
      "We should to him and nothing can I never my\n",
      "brothers forbeard our words:\n",
      "Confess which have wine, and provide thee\n",
      "And see how did bestride our nobising duke,\n",
      "Whose souls more pleased, that men are worst challenge into a farmet together\n",
      "Let hate not so, for these own with him.\n",
      "\n",
      "Tailors:\n",
      "But now the blood of two,\n",
      "You and your accused find your wars.\n",
      "\n",
      "MENENIUS:\n",
      "Part not chastisement; hidest me well arm'd\n",
      "With eterpity of the mon poor house.\n",
      "\n",
      "Third Servant:\n",
      "When you are gentlemen, some interr'd\n",
      "As is my goods, God says King Richard say,\n",
      "As is the sin that she doth aim at you\n",
      "As I could pick my daughter woful side!\n",
      "Thou follow'st me in his life, against your husband's land:\n",
      "My lawful king is dead! wis gone to ET:\n",
      "Where is my woful? Was never man so bath\n",
      "Lord Her further.\n",
      "\n",
      "BRUTUS:\n",
      "Let me prepare from fair, nor ambition\n",
      "That spucish him our absence, like a grave is no more drea\n"
     ]
    }
   ],
   "source": [
    "# Generate the text with default temperature (1.0).\n",
    "print(generate_text(model, start_string=u\"ROMEO: \"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 561
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1208506,
     "status": "ok",
     "timestamp": 1584521016280,
     "user": {
      "displayName": "Oleksii Trekhleb",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GiiA4aUKCbFho88Jd0WWMoAqQUt3jbuCtfNYpHVOA=s64",
      "userId": "03172675069638383074"
     },
     "user_tz": -60
    },
    "id": "wq_NlwWJSdix",
    "outputId": "408ecde5-0646-4eda-902e-071e4f123417"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ROMEO: hexen his virginallike.\n",
      "\n",
      "BRUTUS:\n",
      "For thyself\n",
      "Look, sirr; let not this innocent mine friends\n",
      "And hateful dogs as appeared, for it exced\n",
      "Exelling clouds with gentle knterpleaking:\n",
      "Their liberal dogs not his;\n",
      "For Yorkome help, ho! ay, I trow:\n",
      "it will we meet us king E:\n",
      "Needs friends, father, in that mad bessisking in\n",
      "Whom I might thrive is rich'd die by thee?\n",
      "\n",
      "CORIOLANUS:\n",
      "Brak how, gnow roaring with Sicils\n",
      "Up Jusul's dee Signior Katharina,\n",
      "And thyself shalt hear of that which the ear o' Lord:\n",
      "You are the first gape on me at first,\n",
      "Thou lovest me wash:\n",
      "By heaven, I'll haughthere.\n",
      "O, gentle marvantio; and beginn pluck'd\n",
      "But the christs and Riciards a\n",
      "LUUMEOfoke fray,\n",
      "And yet whick, in my doesder differences be runed by the king,\n",
      "Who in his monthry glorious regeal death.\n",
      "O, that she smooth thy cause?\n",
      "Was e turn be so\n",
      "to thine as thee won that Edward, king, widdwisk\n",
      "Ere te goil: deserve not thy mother leave for reason\n",
      "many an hourable fought, as they do,\n",
      "If you should esear mon-judgment,\n",
      "Nor \n"
     ]
    }
   ],
   "source": [
    "# Generate the text with higher temperature to get more unexpected results.\n",
    "print(generate_text(model, start_string=u\"ROMEO: \", temperature=1.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0hh80MqEO_XI"
   },
   "source": [
    "## Save the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VPE98xa8PA-u"
   },
   "outputs": [],
   "source": [
    "model_name = 'text_generation_shakespeare_rnn.h5'\n",
    "model.save(model_name, save_format='h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WYP08xbbTNKp"
   },
   "source": [
    "## Converting the model to web-format\n",
    "\n",
    "To use this model on the web we need to convert it into the format that will be understandable by [tensorflowjs](https://www.tensorflow.org/js). To do so we may use [tfjs-converter](https://github.com/tensorflow/tfjs/tree/master/tfjs-converter) as following:\n",
    "\n",
    "```\n",
    "tensorflowjs_converter --input_format keras \\\n",
    "  ./experiments/text_generation_shakespeare_rnn/text_generation_shakespeare_rnn.h5 \\\n",
    "  ./demos/public/models/text_generation_shakespeare_rnn\n",
    "```\n",
    "\n",
    "You find this experiment in the [Demo app](https://trekhleb.github.io/machine-learning-experiments) and play around with it right in you browser to see how the model performs in real life."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "text_generation_shakespeare_rnn.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}