Chapter04/ch4_nb5_explore_imagenet_and_its_tiny_version.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style=\"border: 1px solid #e7692c; border-left: 15px solid #e7692c; padding: 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c\">Tip.</strong> <a style=\"color: #000000;\" href=\"https://nbviewer.jupyter.org/github/PacktPublishing/Hands-On-Computer-Vision-with-Tensorflow/blob/master/ch4/ch4_nb5_explore_imagenet_and_its_tiny_version.ipynb\" title=\"View with Jupyter Online\">Click here to view this notebook on <code>nbviewer.jupyter.org</code></a>. \n",
    "    <br/>These notebooks are better read there, as Github default viewer ignores some of the formatting and interactive content.\n",
    "    </p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table style=\"font-size: 1em; padding: 0; margin: 0;\">\n",
    "    <tr style=\"vertical-align: top; padding: 0; margin: 0;background-color: #ffffff\">\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; padding-right: 15px;\">\n",
    "    <p style=\"background: #363636; color:#ffffff; text-align:justify; padding: 10px 25px;\">\n",
    "        <strong style=\"font-size: 1.0em;\"><span style=\"font-size: 1.2em;\"><span style=\"color: #e7692c;\">Hands-on</span> Computer Vision with TensorFlow 2</span><br/>by <em>Eliot Andres</em> & <em>Benjamin Planche</em> (Packt Pub.)</strong><br/><br/>\n",
    "        <strong>> Chapter 4: Influential Classification Tools</strong><br/>\n",
    "    </p>\n",
    "\n",
    "<h1 style=\"width: 100%; text-align: left; padding: 0px 25px;\"><small style=\"color: #e7692c;\">\n",
    "    Notebook 5:</small><br/>Exploring ImageNet and Tiny-ImageNet</h1>\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #363636; text-align:justify; padding: 0 10px;\">\n",
    "    In this additional notebook, we demonstrate how those interested can acquire <em><strong>ImageNet</em></strong> and its smaller version <em><strong>Tiny-ImageNet</em></strong>, and can set up training pipelines using them. With this notebook, we will also briefly introduce the <code>tf.data</code> API.\n",
    "</p>\n",
    "<br/>\n",
    "<p style=\"border-left: 15px solid #e7692c; padding: 0 10px; text-align:justify;\">\n",
    "    <strong style=\"color: #e7692c;\">Tip.</strong> The notebooks shared on this git repository illustrate some notions from the book \"<em><strong>Hands-on Computer Vision with TensorFlow 2</strong></em>\" written by Eliot Andres and Benjamin Planche, published by Packt. If you enjoyed the insights shared here, <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Learn more about the book!\"><strong>please consider acquiring the book!</strong></a>\n",
    "<br/><br/>\n",
    "The book provides further guidance for those eager to learn about computer vision and to harness the power of TensorFlow 2 and Keras to build efficient recognition systems for object detection, segmentation, video processing, smartphone applications, and more.</p>\n",
    "        </td>\n",
    "        <td style=\"vertical-align: top; padding: 0; margin: 0; width: 280px;\">\n",
    "    <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Learn more about the book!\" target=\"_blank\">\n",
    "        <img src=\"../banner_images/book_cover.png\" width=280>\n",
    "    </a>\n",
    "    <p style=\"background: #e7692c; color:#ffffff; padding: 10px; text-align:justify;\"><strong>Leverage deep learning to create powerful image processing apps with TensorFlow 2 and Keras. <br/></strong>Get the book for more insights!</p>\n",
    "    <ul style=\"height: 32px; white-space: nowrap; text-align: center; margin: 0px; padding: 0px; padding-top: 10px;\">\n",
    "    <li style=\"display: block;height: 100%;float: left;vertical-align: middle;margin: 0 25px 10px;padding: 0px;\">\n",
    "        <a href=\"https://www.amazon.com/Hands-Computer-Vision-TensorFlow-processing/dp/1788830644\" title=\"Get the book on Amazon (paperback or Kindle version)!\" target=\"_blank\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_amazon.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    <li style=\"display: inline-block;height: 100%;vertical-align: middle;float: right;margin: -5px 25px 10px;padding: 0px;\">\n",
    "        <a href=\"https://www.packtpub.com/application-development/hands-computer-vision-tensorflow-2\" title=\"Get your Packt book (paperback, PDF, ePUB, or MOBI version)!\" target=\"_blank\">\n",
    "        <img style=\"vertical-align: middle; max-width: 72px; max-height: 32px;\" src=\"../banner_images/logo_packt.png\" width=\"75px\">\n",
    "        </a>\n",
    "    </li>\n",
    "    </ul>\n",
    "        </td>\n",
    "        </tr>\n",
    "        </table>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import glob\n",
    "import tensorflow as tf\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tiny-ImageNet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Presentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As presented in the chapter, the *ImageNet* dataset ([http://image-net.org](http://image-net.org)) and its yearly competition pushed forward the development of performant CNNs for image recognition[$^1$](#ref).\n",
    "While it could have been interesting to reuse this dataset to reproduce the results listed in the book, its huge size makes _ImageNet_ difficult to deploy on most machines (memory-wise). Training on such a dataset would also be a long, expensive task.\n",
    "\n",
    "Another solution could have been to use only a portion of _ImageNet_. Indeed, the people at Standford University already compiled such a dataset for one of their famous classes (\"_CS231n: Convolutional Neural Networks for Visual Recognition_\" - http://cs231n.stanford.edu/). This dataset, _Tiny-ImageNet_ ([https://tiny-imagenet.herokuapp.com](https://tiny-imagenet.herokuapp.com)) contains 200 different classes (against the 1,000 of ImageNet). For each class, it offers 500 training images, 50 validation images, and 50 test ones."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tiny-ImageNet can be downloaded at [https://tiny-imagenet.herokuapp.com](https://tiny-imagenet.herokuapp.com) or [http://image-net.org/download-images](http://image-net.org/download-images) (users need the proper access).\n",
    "\n",
    "***Note:*** Makee sure to check the _ImageNet_ terms of use: [http://image-net.org/download-faq](http://image-net.org/download-faq).\n",
    "\n",
    "Once downloaded, the archive can be unzipped (`unzip tiny-imagenet-200.zip`) at a proper location. Its path is stored into a variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "ROOT_FOLDER = os.path.expanduser('~/datasets/tiny-imagenet-200/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let us have a look at the directory structure of the dataset:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "    - <ROOT_FOLDER>/tiny-imagenet-200/\n",
    "       - wnids.txt                 <-- File with the list of class IDs in the dataset\n",
    "         \n",
    "       - words.txt                 <-- File with the mapping from class IDs to readable labels\n",
    "         \n",
    "       - train/                    <-- Training folder\n",
    "          - <class_i>/             <--    Folder containing training data of class <class_i> \n",
    "             - images/             <--       Sub-folder with all the images for this class\n",
    "                 - ***.JPEG\n",
    "             - n01443537_boxes.txt <--    Annotations for detection tasks (unused)\n",
    "                         \n",
    "       - val/                      <-- Validation folder\n",
    "          - images/                <--    Folder with all the validation images\n",
    "          - val_annotations.txt    <--    File with the list of eval image filenames and\n",
    "                                          the corresponding class IDs\n",
    "         \n",
    "       - test/                     <-- Test folder\n",
    "          - images/                <--    Folder containing all the test images"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we define some additional dataset-related constants useful for later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGENET_IDS_FILE_BASENAME = 'wnids.txt' # File in ROOT_FOLDER containing the list of class IDs\n",
    "IMAGENET_WORDS_FILE_BASENAME = 'words.txt' # File in ROOT_FOLDER containing the mapping from class IDs to readable labels\n",
    "IMG_HEIGHT, IMG_WIDTH, IMG_CHANNELS = 64, 64, 3 # Image dimensions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Datasets come in all forms and sizes. As training a CNN is a complex and heavy process, it is important to have an efficient data pipeline to provide the training batches on time to avoid performance bottlenecks.\n",
    "\n",
    "In the following section, we will set up an input pipeline for a Tensorflow model, using Tiny-ImageNet as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing the Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_Tiny-ImageNet_ is mainly organized by class. Therefore, let us start by listing and parsing those various classes.\n",
    "\n",
    "We will use the two text files at the root of _Tiny-ImageNet_ to:\n",
    " - List the IDs corresponding to the 200 classes. This list will allow us to assign to each ID (IDs are 9-character-long strings) an integer from 0 to 199 (the ID position in the list);\n",
    " - Build a dictionary to map the IDs to human-readable labels (e.g., '_n01443537_' $ \\rightarrow$ '_goldfish, Carassius auratus_')\n",
    "\n",
    "The first list is the most important, as it defines the categories (mapping the string IDs to numbers) which will be the target of our recognition models. The second structure, the dictionary, will simply allow us at the end to get understandable results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _get_class_information(ids_file, words_file):\n",
    "    \"\"\"\n",
    "    Extract the class IDs and corresponding human-readable labels from metadata files.\n",
    "    :param ids_file:        IDs filename (contains list of unique string class IDs)\n",
    "    :param words_file:      Words filename (contains list of tuples <ID, human-readable label>)\n",
    "    :return:                List of IDs, Dictionary of labels\n",
    "    \"\"\"\n",
    "    with open(ids_file, \"r\") as f:\n",
    "        class_ids = [line[:-1] for line in f.readlines()] # removing the `\\n` for each line\n",
    "\n",
    "    with open(words_file, \"r\") as f:\n",
    "        words_lines = f.readlines()\n",
    "        class_readable_labels = {}\n",
    "        for line in words_lines:\n",
    "            # We split the line between the ID (9-char long) and the human readable label:\n",
    "            class_id = line[:9]\n",
    "            class_label = line[10:-1]\n",
    "\n",
    "            # If this class is in our dataset, we add it to our id-to-label dictionary:\n",
    "            if class_id in class_ids:\n",
    "                class_readable_labels[class_id] = class_label\n",
    "\n",
    "    return class_ids, class_readable_labels\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can directly test this function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "ids_file = os.path.join(ROOT_FOLDER, IMAGENET_IDS_FILE_BASENAME)\n",
    "words_file = os.path.join(ROOT_FOLDER, IMAGENET_WORDS_FILE_BASENAME)\n",
    "class_ids, class_readable_labels = _get_class_information(ids_file, words_file)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\"n02124075\" --> \"Egyptian cat\"\n",
      "\"n04067472\" --> \"reel\"\n",
      "\"n04540053\" --> \"volleyball\"\n",
      "\"n04099969\" --> \"rocking chair, rocker\"\n",
      "\"n07749582\" --> \"lemon\"\n",
      "\"n01641577\" --> \"bullfrog, Rana catesbeiana\"\n",
      "\"n02802426\" --> \"basketball\"\n",
      "\"n09246464\" --> \"cliff, drop, drop-off\"\n",
      "\"n07920052\" --> \"espresso\"\n",
      "\"n03970156\" --> \"plunger, plumber's helper\"\n"
     ]
    }
   ],
   "source": [
    "# Let's for example print the 10 first IDs and their human-readable labels:\n",
    "for i in range(10):\n",
    "    id = class_ids[i]\n",
    "    print('\"{}\" --> \"{}\"'.format(id, class_readable_labels[id]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Listing All Images and Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the categories defined, we can list all the images along with their respective categorical labels.\n",
    "\n",
    "Since the dataset structure is different for training/validation/testing splits, we have to cover them separately. This happens often in practice, as defining a normalized structure for datasets is a complicated task (image format, annotation types, folder structure, etc. are heavily affected by the use-cases).\n",
    "\n",
    "In this example, we will cover only the training and validation split:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _get_train_image_files_and_labels(root_folder, class_ids):\n",
    "    \"\"\"\n",
    "    Fetch the lists of training images and numerical labels.\n",
    "    We assume the images are stored as \"<root_folder>/train/<class_id>/images/*.JPEG\"\n",
    "    :param root_folder:     Dataset root folder\n",
    "    :param class_ids:       List of class IDs\n",
    "    :return:                List of image filenames and List of corresponding labels\n",
    "    \"\"\"\n",
    "    image_files, labels = [], []\n",
    "\n",
    "    for i in range(len(class_ids)):\n",
    "        class_id = class_ids[i]\n",
    "        # Grabbing all the image files for this class:\n",
    "        class_image_paths = os.path.join(root_folder, 'train', class_id, 'images', '*.JPEG')\n",
    "        class_images = glob.glob(class_image_paths)\n",
    "        # Creating as many numerical labels:\n",
    "        class_labels = [i] * len(class_images)\n",
    "\n",
    "        image_files += class_images\n",
    "        labels += class_labels\n",
    "\n",
    "    return image_files, labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _get_val_image_files_and_labels(root_folder, class_ids):\n",
    "    \"\"\"\n",
    "    Fetch the lists of validation images and numerical labels.\n",
    "    We assume the images are stored as \"<root_folder>/train/<class_id>/images/*.JPEG\"\n",
    "    :param root_folder:     Dataset root folder\n",
    "    :param class_ids:       List of class IDs\n",
    "    :return:                List of image filenames and List of corresponding labels\n",
    "    \"\"\"\n",
    "    image_files, labels = [], []\n",
    "\n",
    "    # The file 'val_annotations.txt' contains for each line the image filename and its annotations.\n",
    "    # We parse it to build our dataset lists:\n",
    "    val_annotation_file = os.path.join(root_folder, 'val', 'val_annotations.txt')\n",
    "    with open(val_annotation_file, \"r\") as f:\n",
    "        anno_lines = f.readlines()\n",
    "        for line in anno_lines:\n",
    "            split_line = line.split('\\t')   # Splitting the line to extract the various pieces of info\n",
    "            if len(split_line) > 1:\n",
    "                image_file, image_class_id = split_line[0], split_line[1]\n",
    "                class_num_id = class_ids.index(image_class_id)\n",
    "                if class_num_id >= 0: # If the label belongs to our dataset, we add them:\n",
    "                    image_files.append(image_file)\n",
    "                    labels.append(class_num_id)\n",
    "\n",
    "    return image_files, labels\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we call the method for the training data, we obtain our list of 500 * 200 = 100,000 images and their labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training images: 100000\n"
     ]
    }
   ],
   "source": [
    "image_files, image_labels = _get_train_image_files_and_labels(ROOT_FOLDER, class_ids)\n",
    "print(\"Number of training images: {}\".format(len(image_files)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Building an Iterable Dataset with Tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to convert this list of filenames into images, and generate a list of batches our model could iterate over during its training. There are however lots of elements to take into consideration. \n",
    "\n",
    "For instance, pre-loading all the images may not be possible for modest machines (at least for bigger datasets); but loading images on the fly would cause continuous delays. Also, in several papers we presented in Chapter 4, data scientists are applying random transformations to the images at each iteration (cropping, scaling, etc.). Those operations are also consuming.\n",
    "\n",
    "All in all, we would probably need some multi-thread pipeline for our inputs. Thankfully, Tensorflow provides us with an efficient solution. Its **`tf.data`** API contains several methods to build **`tf.data.Dataset()`** instances, a dataset structure which can be converted into batch iterators for TF models.\n",
    "\n",
    "***Note:*** The `tf.data` API is thoroughfully detailed later in Chapter [7](./ch7)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For instance, a `Dataset` can be created from tensors containing lists of elements. Therefore, we can easily wrap our `image_files` and `image_labels` into a `Dataset`, first converting them into tensors: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<TensorSliceDataset shapes: ((), ()), types: (tf.string, tf.int32)>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image_files = tf.constant(image_files)\n",
    "image_labels = tf.constant(image_labels)\n",
    "dataset = tf.data.Dataset.from_tensor_slices((image_files, image_labels))\n",
    "dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This object has multiple methods to transform its content, batch the elements, shuffle them, etc. Once defined, those operations will be applied only when necessary / called by the framework (like any other operation in TF graphs).\n",
    "\n",
    "Our goal is to have this dataset output batches of images and their labels. So first thing first, let us add an operation to obtain the images from the filenames:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _parse_function(filename, label, size=[IMG_HEIGHT, IMG_WIDTH]):\n",
    "    \"\"\"\n",
    "    Parse the provided tensors, loading and resizing the corresponding image.\n",
    "    Code snippet from https://www.tensorflow.org/guide/datasets#decoding_image_data_and_resizing_it (Apache 2.0 License).\n",
    "    :param filename:    Image filename (String Tensor)\n",
    "    :param label:       Image label\n",
    "    :param size:        Size to resize the images to\n",
    "    :return:            Image, Label\n",
    "    \"\"\"\n",
    "    # Reading the file and returning its content as bytes:\n",
    "    image_string = tf.io.read_file(filename)\n",
    "    # Decoding those into the image\n",
    "    # (with `channels=3`, TF will duplicate the channels of grayscale images so they have 3 channels too):\n",
    "    image_decoded = tf.io.decode_jpeg(image_string, channels=3)\n",
    "    # Converting to float:\n",
    "    image_float = tf.image.convert_image_dtype(image_decoded, tf.float32)\n",
    "    # Resizing the image to the expected dimensions:\n",
    "    image_resized = tf.image.resize(image_float, size)\n",
    "    return image_resized, label\n",
    "\n",
    "dataset = dataset.map(_parse_function)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "`dataset.map(fn)` tells the dataset to apply the function `fn` to each element requested at a given iteration. These functions can be chained. For example, we can add another function to randomly transform the training images, to artificially increase the number of different images our model can train on:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<MapDataset shapes: ((64, 64, 3), ()), types: (tf.float32, tf.int32)>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def _training_augmentation_fn(image, label):\n",
    "    \"\"\"\n",
    "    Apply random transformations to augment the training images.\n",
    "    :param images:      Images\n",
    "    :param label:       Labels\n",
    "    :return:            Augmented Images, Labels\n",
    "    \"\"\"\n",
    "\n",
    "    # Randomly applied horizontal flip:\n",
    "    image = tf.image.random_flip_left_right(image)\n",
    "\n",
    "    # Random B/S changes:\n",
    "    image = tf.image.random_brightness(image, max_delta=0.1)\n",
    "    image = tf.image.random_saturation(image, lower=0.5, upper=1.5)\n",
    "    image = tf.clip_by_value(image, 0.0, 1.0) # keeping pixel values in check\n",
    "\n",
    "    # Random resize and random crop back to expected size:\n",
    "    original_shape = tf.shape(image)\n",
    "    random_scale_factor = tf.random.uniform([1], minval=0.7, maxval=1.3, dtype=tf.float32)\n",
    "    scaled_height = tf.cast(tf.cast(original_shape[0], tf.float32) * random_scale_factor, \n",
    "                            tf.int32)\n",
    "    scaled_width = tf.cast(tf.cast(original_shape[1], tf.float32) * random_scale_factor, \n",
    "                           tf.int32)\n",
    "    scaled_shape = tf.squeeze(tf.stack([scaled_height, scaled_width]))\n",
    "    image = tf.image.resize(image, scaled_shape)\n",
    "    image = tf.image.random_crop(image, original_shape)\n",
    "\n",
    "    return image, label\n",
    "\n",
    "dataset.map(_training_augmentation_fn)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also specify if we want the dataset to be suffled, or sepcify how many elements we want at each iteration in a batch, how many times we want the dataset to be repeated (for multiple epochs), how many batches to pre-fetch, etc:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 32\n",
    "num_epochs = 30\n",
    "\n",
    "dataset = dataset.shuffle(buffer_size=10000)\n",
    "dataset = dataset.batch(batch_size)\n",
    "dataset = dataset.repeat(num_epochs)\n",
    "dataset = dataset.prefetch(1)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***Note:*** More detailed explanations on `Dataset` and its methods, as well as performance recommendations, will be provided in Chapter 7 and its [notebooks](../ch7)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our dataset is ready, and we can now simply iterate over it to obtain our batches:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "images, labels = next(dataset.__iter__())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "organ, pipe organ\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<matplotlib.image.AxesImage at 0x7f1dae6e6550>"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAD8CAYAAABXXhlaAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJztvXmUpFd1J/h7EZEZERkRuS+1ZFVlbZKqVNpAYCEWrWCxGLpnwG2M27QP58jtNnOg7WkDPZs9p2cGxmdsfGbGHNS2u6GHZjGbaBqzCYSbBoRKFJKqVJJqX7Jy3zNjj3jzR0Z993dvVWalkCpKON7vnDr1It+LL1583/fiu/f97v1d571HQEBAayF2rScQEBDQfISFHxDQgggLPyCgBREWfkBACyIs/ICAFkRY+AEBLYiw8AMCWhAvauE75x5wzj3nnDvunPvwSzWpgICAqwv3iwbwOOfiAJ4H8EYA5wE8DuDd3vtnXrrpBQQEXA0kXsR7Xw3guPf+JAA45z4H4B0A1lz47e3tPpVOAQBSyaTqq1QrUdv7uurzdflxisfFSKnX9ThAxrUl2lRPvpCP2plMR9Qul8tqXFtCTon9UXTOXXaOdlw85rAWeF4FmhMAdHd3U18hai+vrKx5vFpdf3atJq9rqssYd/Rd4NY2/Pi71GpVfQhIX1ubPt+lkpzXWCwux4vrWy4Wk89ub2/XxyiW5Pjtcvx6vWZmKV+0oyOtemo8lr8y9DWqe7539Bz5+sZj+lzxPajvOX1dEtS3tLS09vHps9Mp/V3qdMxku14/5crq+pmdmcHK8vLaN+DF+VxpwDrYCuAcvT4P4FfWe0MqncIdd94OANizZ4/qGx+/ELWrVb0YyyW5AXI5WbSFvF44gNyYmwYHVc9TTx2K2q+6/faoPTp6To3r75XFZ39Y2trkBq5WZE6Vkp5vJi03sDMG1ZZN/VH7yOGnVd/b3/bWqH348OGo/aOfPKYPQgt1Pl9RXQtLMpfFsnx4HR1qnG9LyYt4Chryvt5OuWHnZ6fVqERcvuemoa2q7/ixs1G7oyMXtbu7+tW4TCYbtbdt26aPcfyY9G3fHLWXlxfUOBeTc3DrbTepvsWleRmXkPUQj+mFUygVo/aWwQHVV6nI8bs6sqovT/fgpgG552o1/eM0OLApan/v0e+rvlpN7rNcj9x/B268VY0r1eX+3rFjp+obvbC6fv78//goNoIXs/Av96tyid/gnHsQwIMAkEolL3lDQEBA8/FiFv55APwTPQzggh3kvX8IwEMAkMl2+Lm5OQDA8vKiGleip3pfX4/qSyXlqcMW6oQx08+fPx+1O7P6l3l5WczlY8eOUY9+qs+Q6ZzrzKi+TEZeH3lanshxYylfv1esmXe849dUX4nM++v2aqvn6BE5Jj8xbrvtNjXu2PFTUTvWrp8sCXodz8sTYrmkf5NLZLXXqtpq4HNSovetGJejq1Oe+NZ8TaXZbKcnWk6f02RSrI1yuaT6Bgb6ovb27cNR++Spghp30023RO1njmorKk5P+QMHDkTt2Xl9/8XJirJP654ueQpnjPmdIpObLcRisajGPfXUU1E7l9H3ZrkqF6OXPquvX6+Ds+dGo/bhw0+pvv6GleKuaOSv4sXs6j8OYK9zbqdzrh3AbwD42os4XkBAQJPwCz/xvfdV59z7AXwLQBzA33jvj7xkMwsICLhqeDGmPrz33wDwjZdoLgEBAU3Ci1r4LxTe+2jH3hlnJEZ7hd7sprNvWamIX5/L5dS4/n7ZMU4YSmbrVtl1TqdTNE57O+Ojsk2RTOmdaj5+JtcZtctF7XOeOSfH+MpXHlZ9bbQh0G7Ofi4r/m+5Kn6mpRzZl4wZeqmtLUZtoolK2o/35NfXvKXHBCXaJyitaDrPZ+T4y8bHT5PvXqnI8XNZzSC0E023e9ew6jt3ThiXM6dPRO1MWh+DP3t4iz4GX+t4TD7r23/3TTUulZJxw1v0db/15pujdjGhr3UH+fylmlynhLkuKaIqOzo0w8JU39SMMCdz01Nq3OTEWNRO22OsQyFfDiFkNyCgBREWfkBAC6Kppn69Voton0JBU0Nt7RIcY034Wk3M0ulpMYWsu7CyImZY955u1be0JEEfE2PjUbs9qSPOMhn57N279qq+6anZqH36lASo2Ki1In23O++8U/V5Mgf7evQcDz3xeNRm2mtiUpt8BTLbK1VN0xXJrKZ4j0sox3ZyCeL1uOqrUlRijJ4N6aSOrIvRR1uXqZ1eZ9JCeaWS+prlcmJiH3z8v6q+17/+9VH7wgWhaqtV7XIQY4e6iUK8QBRY9ay4DpZSGxiQoB0b/LV9WFjrmSkdxJSkCNQSRVtmDZ3c1yfU5MmTJ1Xf5h0S3DM1Jdc6ndIuzbatMm5k127VNzGxek+7S0NpLovwxA8IaEGEhR8Q0IIICz8goAXRVB+/rb0Nw1tXky048QYAFhclhJKTYQCgr6c3andmxQdPmHEc9nvq1CnVx2HAHII5NT2pxnFCxlJeUzeP/0wSfZId4oN3d+nQyhlKZhkd0/55NiM+Yb44ofoe+cHBqD00IHTN29/+djXupz+VcYt5HRrKsbjOczaX9s8TbXLpy2afoFqljDN6NiS6etU4D9lP6DT7Mky7dneRv+ttNqR89t49O1Tf8WOS6Llpk/i3o6OjatxEWc5Bb6+e47v+21+P2p/+9KdlTtlONS6bkvNdMdRnflnCrGdn51Rfwsk9OD8nfYUVfe/s378vaqeM7x6LyybFpk1DcmxDCS4vyj7VE4/pxK2VwjIAoFgw98MaCE/8gIAWRFj4AQEtiKaa+vFYHJ2dqyZWPr9sesW8HB8fVz15yqxjM6nXZPFtoait4c2bVF/diwk8OS4m9tSkpmd27BBzszPXpfpWyOSLk6k8NqHdhV27dkVtjuIDgJ07ZI65rI6+2r13e9R+7qjQhfE2baZ39wo1VI/p3PRiSczvCuka2LguR+e7YiL36hU5V1UvVOUlYhtlGWefIJ64xHhMPquQn1fjSh3kcpgIRc6SGx8T87u7S1+XAzdKZF06rWm06cmZqL20IPecTRHnyNFyUbuhBbr/OpLaTN80JKY5C3jY7DwW27huj6aJ2c2Nk4DDz392UI37yeM/idrW3fngBz8IQEdCrofwxA8IaEGEhR8Q0IJoqqlfLBbx/LNHAVyqU7dvn+x6rhg3oES7trU6m5cmSokkjGZmZlQXuwh1L2adMxpwoxTVNzuvzWjWsBvql+iuM+e0+EM6I5+9f/8Nqu/RH/4oat+073rVNz4hDMCNN4m7ML+ozwfPua1Nm6ysW1euyIS92blnM9rV9S42R+RxlFzC6XOVIhekVtHHyLH2HR0/Zm65WlXcp55uvdM+PycmcDuF5/V26XGP/fjHUXt4q2YGHMT0rRDr05nV7kInJUhZl6ZOWoNV8z1LxPxUSYJtydw77C4sL+vruZKXJKNTZ85EbRYRAYA8idckDaPV07XKqsRtiOYaCE/8gIAWRFj4AQEtiLDwAwJaEE318Z2TjLqJSU3Z9ZOwImfgAVrkcojoE5ulFY+L35M30ttMr3Df0OBmNc6T/z8/P6v6OnMU8TcjPtzr33CPGvfEE09E7fPnNZ2XyQkF+eTTugRBFx1/z17e89D0Egtb2Fws9vGTdHocNGXnXJ3a2pdMJuQ8FvIyzoqb5DrFj6/WjFDmoFzPcln84M4uTbclKTvSUrwkx48sUZ9j45rKesMb7o7avq6fZVOTEk3Hwph5Ixy6ZbPs2fA9BmjhjJKJ5mQKkjP1bHZelYRVvve976m+oU2SGXiERDTv/9U3qXH9ROOeNfsEH/pX/woAML+k18RaCE/8gIAWRFj4AQEtiCab+i4Sc8ga3bTFOTGr6xUdwZVNi4lz5qRor8Xjmna59VapPDIxoTXgenrExJ6YFrpt2SQ1ZKhSSk+/rqhyhqiWdqLRJie0a5KliL/RCzoRJ5WUUx4zZb5uvFn04Y+fkCQjq6++SPSeNyF5y5QcEotRJFlJuz4p+p7FojYba1V5HvgauT5Lelx3t0QhJtv19Vyhsdt3iIad95oOq1PUoBWRyBIlyFWTbiEdfQDIUUm0qSntnq0QBcYJWNuGdUWfLhI+mTWJW7kd0mc19zkxbPd1EpHHOvoAsEz1FPqHtNDHLbdI5OGmTdIXt/GWRFdb+vTi/JdWbETs5RGe+AEBLYiw8AMCWhBh4QcEtCCa6uNXK5WoKq71lVgr3tJ03MfUXrmsx42Nie74hQuaRuMwzDSJLtjQ3tkZoX/SaV3nrUohsHESk7TCCrmq+M+FmA6t3LJZ9g2WFrQ/Ojcv+xJ5Cv/srWtfr0x0XtKUG+dz198vohQrK9rH7+mUfYiFBb0fwuKmWaIYu0xWHNOAVt+/t1f2JWZnqGJtTF/3+++/N2r/8Ic/VH3ViuxXDA3KPgHv1wDad5+Y0HsqF7NBAeDATRI+3Z7Q+wlM542N62Pw+bBU4t4bJOz6yBEpJLVn93Vq3Gc/95mofejQIdU3sk2yMgcGhEpcXNSZjEwRxoz/f3FPpXZJCfHL44pPfOfc3zjnJp1zh+lvvc657zjnjjX+71nvGAEBAS8vbMTU//cAHjB/+zCAR7z3ewE80ngdEBDwS4Irmvre+793zo2YP78DwN2N9qcAPArgQ1c6VmdnDm+8/3UALhUq4DLLzzyjI9omyXzj6DE2eQFdastm3XFm3aZNW6K2qdalMqdsXzZHEVxE40xPa5OdRShsaemeHjEBfU27Kgtk6re1C5XV1aOpp6eOyPlh1wcAJmdkLtlOMcSKRuQiRkIi1u3ixMlzo6JF39+n6c32NBl6xqWpkG4fl4GuG1P0/Khc28EhXbrqHGniT8/K95qd05lv/Pw6T/r7ADCyY2fU7u4WV6Ve09dlaVlcvFhcuwHdlDWYymi3jjXy77jjNVH72eeeU+P27ZcS3aeJFgYAUMQpiILV5dy1YIe97un06v0yX9DU8lr4RTf3hrz3YwDQ+H/wCuMDAgJeRrjqm3vOuQcBPAgAmY7UFUYHBAQ0A7/owp9wzm323o855zYDmFxroPf+IQAPAUB/X1dkQ9kdYt6xtCIdNbLTeSd5cUmba3Mkb9zTp83jZdLLSyTEVCwZKeV2ikDLF3TiCSfwcPRc2uiwcSXThQUtx+xIjrlotN3GxoSJ6KAfSbtz39UtJrZlFCrEdLBWX9bISbPLxFWAAS1UMjAobpGNlOT32QSb+XkxS2+7TSLtRkZG1DjeJR/eulP1dXeJa8HXll01QJ9T63Yx+7J1i+gwTo5rc7tQkPlXrdw45B5JmpJrc8TMeNLL+94PHlXjHvuRyGFzMhmgozTf9Mb7ojZHVwJAitioek3PcWJy1WVi5mk9/KKm/tcAvLfRfi+Ah9cZGxAQ8DLDRui8zwL4MYDrnXPnnXPvA/BRAG90zh0D8MbG64CAgF8SbGRX/91rdN23xt8DAgJe5mhq5F6lUsHUxGp0nfXxWUBiy5DWxC+SzjtHiPX2aX+GSxGnTRlkpoamyA+00X8XaREAKJUMBUaUVSol48oFLc6Qy5DQZE1zgqxZXzEZVizqUCehjFOG/tm+XSK9LBWXyYkvP0binWWThbhAexTWd+d9FEdlp8tV/V3aUvI+Sy/x0JWCfM/lvD6nM7OyT3PkGV0++uabb4ra+WKV2voYHK04t6j3VCaPi+BLuuOVUdvF9HVPpuTaVlb03svMjJzHTE5/z1tf8aqo/V0S2LBZk6+/+66o/cCvvkX1MX2domhRG1HJZeB5XwOQ7EgXSmgFBASshbDwAwJaEE019ROJRJRgYSkqNnPTaW3GzC9RiSEyS9uM+AMn4jB9B2jzNZGQz+7s7Fbj4hTxt2SEJ4pE/XF12PEFHUlWr7K4hAabpTaxJUuVgKskUGFdgt5eodHKZW3a1UkEhE3idlMtlyMKO7v1OeBzNTkpTG3OUIJMOd50002qLxGXc3yC6KqTJ86pcVu2bIva5ZKJmCM6b3xczO280b1rT8r90t6ub2lHFFutLiY8a9kDQLks90tbUpvzRdIMTGV02bPnj0t03aEnfx61j584rcZtHhIK8gaqnAsA3/3eI1G7g85x1biJ3Dc7q2nLi/UVnNNu0FoIT/yAgBZEWPgBAS2IsPADAloQTafzJhuhhUy9AZoqY/EEACiQb61CXk24LQtxjE9qgQ3WRncxpkW0F87RwtWK9rEWqf5ZJi3H27NLh5pyotrMrN6HKFFWImdbAUCBaMEC+e4dJsdhbEIoqkvmT6Id7CKyrjsAOMoCKxRtaLK8sZeuk70ujuq0TUzqqO081QKYnhJBiT27dS3BbE7EQrbv0PsQ8wuUbUnzzReNj5/KrdnXR/Ua8iQ46mN6P6FQlL5ct5aXmF+Q6+SKmgb8znclaHV6RvYNSuZ8F8tyPtqTadXH4c29vXI+lkxIej+VhT9mai0kG/eji+n3rIXwxA8IaEGEhR8Q0IJoqqkfi8WRaUTUJdo1nbdIJZHHJ7WYAFNKPT3sIujfLRbHYJ00QGdETU7J8WzJYqYEkybrjjOzWOu+zZjbczMyf5tBOE2afiy8AQAsrZciSmn7Dl36maP12lP6PJaK8r27ScBjwVCO6Q659IumDDefu1qN9Q51NCRnVNqyZ4ePPB+1r7tOzPvh7dvUuIM/k3Jj7I7ZOe/ePRK1u3u0IAhpimDM1DHYe93uqF2lWyLm9b3TQVRqpiOn+sbGZmmcnuOdd74uan/r2xK599nPf0aN+9cf+R/lsw2Ny5Rjb59Qq0tL+ppt3SJ6fMsr2k3sS63ej5Y+XgvhiR8Q0IIICz8goAXRVFO/VqthrpF4UDVGSRuZ/ps2a+010E5+G5WusmY675JbcQmW0U6nqKJsUkcJrrAgQ01HxXG13DNnJKEk7vRp7KZIuFyX2SEmZiDWrs3GMu34s5jH+LiWWeZEouE+LUoxNS2Rce3tco4zOX0+2H3KZPUcCwVmL6jC7LLeMR8tCYuSN8xAplNclWJV+gpVHVHpkzLHA6+8VfU98dPHo/bNr5AEm1Mnj6txWzdJxePr9uoIwhkql9bVJd+lO6d31iu06x6Laxcv0S7f5enDR1Xfc89KSbeOjLAeXVntLlTIDWXtRkCLmHTSeYPT7mq6Q+7VgUEdbbnYSE6q1UO13ICAgDUQFn5AQAsiLPyAgBZEU318OBdFYDHtBABlEp7gSCZACxIkk5J9ZEUoOOPPRrQxTcft9nYtQlGpySmxop/livjgTB36ut4nqHI5MCPOvzwr3y2X037gEmUUZigTywqCztIxBjcNq75UUvxH9rttNlqVyl8nU5qmA1Fdi4tCZW0b1vsJHRmZP0emAcDTpP3f2y8UrI2Ye/1dQod96W+/qPre9ra3R+2jz0p5qoTR8Oe9npMnT6s+jrDsytDeS6e+9QsU1Tc9q/dUHn9cKMcJExE6RzT0wqLsjezZswdrIR630Zbil7dTGXUPfX8nKPOwbnx5pgQ3gvDEDwhoQYSFHxDQgmhu5J6LIdlIULAltEjmXSWrAFqkg81v1oMDgDbSkR8c1MV92A1gYQsrcoG6HN8bM71WkXlwtdJaXZtZafqsWRPR1tcnSRi2Uu8gJZQUiOLJZLUpniFNv2VTUXWO9OHq9LtuhTjY2iybxJbCirgFiYS4QuxGADopamlFU6v8eXfdJXpzc7M6Gu3pp5+O2u//vX+u+n70o59E7Z4uMdNT5rv817//QdR+1W2aEjx8OKr1itNnRBAkNamv2XXX74raE1P6uhw4IOWvTnztP6u+L3/5S1H7M//xs1Hbaud7ElaxfeyyMlXr62u7q9bNvejahsi9gICANREWfkBACyIs/ICAFkRTfXznXOT75fM6dJNruVkqjv1zzmyy4hKM9ag+/ixu289mYUwLrqNXNDrv81RPbdCEDo9Nil88vHW77iOfmQUZLCWY6xKff2JiTPXNTMlnDwzJPkeyTfuVS8viaztDWyYSTOfJHoIt+c3wRhiyv0fmz5TaydkTalx+UeYxS/sTADDQL3796ePyvk2DQ2rc7LQIkxz6mb6v5ihkt4eOx1QkoCm8p546rPqefEqoxHvvvV/1JdMS3sv3JtO9FiY5DzUSPuH7tG6uO9OYdm/q4t7XRkm9jZTQ2uac+75z7qhz7ohz7gONv/c6577jnDvW+L/nSscKCAh4eWAjpn4VwB967/cBuAPA7zvn9gP4MIBHvPd7ATzSeB0QEPBLgI3UzhsDMNZoLznnjgLYCuAdAO5uDPsUgEcBfGjdgzmh4CoVbUaznr0tH820HdNLls5js/382bOqj0UjmDLJpHWWVo5KQVlTn803NsOWF3SkoRKomNTCEEMUxTY9Oa76tm2VLLOJCXlfV4/WJ1ygktGFkqZFOfOwncz7/IqO3CsS/ZY1GnB9vWISe1KvGB/X36Wbxg0P6whCfj16Tq5FW1xfszfdLyUYF41WPL9uo8i0dvO4euWtkpFnadxZMvXPnj9D43SZtoOHDkbt48d1KS92u9oSWviEsxx37xbRD5fQrhVTeNaV5dfsLljBDr7nrBvgbc2uK+AFbe4550YA3AbgMQBDjR+Fiz8Og2u/MyAg4OWEDS9851wWwJcAfNB7v3il8fS+B51zB51zB8vlypXfEBAQcNWxoYXvnGvD6qL/jPf+y40/TzjnNjf6NwOYvNx7vfcPee9v997fbhNiAgICrg2u6OO7VQfkrwEc9d7/GXV9DcB7AXy08f/Dl3m7hgfqazBwKqvMUH1r1ZuzPhBTIRzmC2j/n/tseDCHmq7ni2U7ZC9gZIcWkFxaEn/6uj27VN+Z86KQs32bVhqaI989Q1r6lZKeI7Ft6EhrnzNJPigr+tjv2dUpdFYqqX+Q52dlHmuFOq/2aaUadXwqgz49Lc8ES7OePSVqOmdMOfC7Xvf6qD1HNOix546ocX1EHT7z9M91X68Ic65QDcbvf//7atz4tFCJU9P6XHVk5H65487XqD6mWpeopt+l52rtzFF+IHJIOofoAnbPSd/7HBK8EWyEx38tgH8K4Gnn3MWz+q+xuuC/4Jx7H4CzAN71gj45ICDgmmEju/o/xNqx//et8feAgICXMZorxAGHiyZKNqPLMW3dKuby2Nio6lssiYlWKYtJk0jo3yN2I9LGDOXX7Erkl3U0Wi0p5pSN6mNKpo1KeScMv5ToknkVjEDlIGXnrSzqTDWm+s6dE5cg26WFFdlsXDFuURvRop7EMRJmN4ddCec1NbSwIFFshYK4C/v26fLOW0jk8sIFfc3OnhaznSMZu7p0xNzhMxKRd999+jnyzb/7etTes0tcpukpvZ00T1mOe3ftVn2crXjuvNCK2/focWfHxZXoH9BzfNs7/pF8thHpGNklghvs4qVNjYD2tLiQNjsvlZJrwTSdLSXPpv6lGX4vbOM8xOoHBLQgwsIPCGhBNNnUl534HhONtn27JKzYHVHWnKuTPW937lncY85EgTEDwGaSrQCbocg9mMQT3hkvV8SMnpnSYhuczMJmHKC1/3v7dAIPfx8uJ5U2NGiRxpVNMkhnv5ipHRn57PEJfa6W5sXNyBqhjy2bJKptmnb4TbCYckfKRnOvkxKJspTIsmlQf+caMRbf++53Vd8N14kZffb0aTqGLqG1e/feqP33339U9W3dKswJC4J8+gtfUOMGh2Xchz70EdX38Ff/U9S+6cAtqm+F7ontI+Kuzs7pe0IJocTtrj7rPFJ5NCM4UifBF2vqW6bgSghP/ICAFkRY+AEBLYiw8AMCWhBN9fE9gHJ11VFcWtG0yCc++W+jti3p/KtvfKMcg3ygssnimyFhy7e9+S2q7yc/EeHG0VHxTdMmO++2226TeczpOYJoryJFaXWaOmm+JrRLjiLYAO3/7zWU0sGDkiHW0yt7ILZ89Cbal7D1A5eJfmPd/sKpU2rcwID4yUvLOnOPo+t438TSS6DvWYW+Fs8eETELjlAcpX0BAOjrkfMz2N+r+ibGL0Ttm28Rwcu5Gb1/Mzsr1/2WV2gfnEuAP/w18dW3mGzCP/u//zJq/+xnP1N9rJxRNhmbHTnZyyjQPoe9r/icqn0k6GvBe0DWb+d9Knv8i2vBbVBuMzzxAwJaEGHhBwS0IJpO512M/rVRYJ/85Cej9vvf/37V97rXSZmlJ56QckbZDm1iv/pVd0Rta07t3SuUT4WouC5jik9NSFSYjdzjiL/BfjHPikUdPcemuTXXRkZGojaLbQBayCFOgiO2BgHTnfweAJhfEtP2qaeeitqWtuQIsc2bN6u+KdKV5/e1Gd0+FgEZNuW16jU5xzt3jsjcTcIRU4njJmKTzVnW3993w41q3Py8uGTO6Tn++Mfi4s0tymff9WadbMM1GThaEQDS2cxl24A24ZmOrZhstDRRmpaKW0urr1rTFCx/lq2YFei8gICAKyIs/ICAFkRY+AEBLYim+vjVSjXyH7u7dcbZAglWWh/r//zYn0btm24WWifVrukl9kdZox7QIoxbNt132b8DwBSJY1ofPH9M/Gem5Ww57TpRKrZvdFT82H37ta/Kc+ZwTSs4skI05rkL+ntyiDB/t3xB023j4yL0aTXmuSz5nt0Svjt2XvvgPXQNR8+dV32dGfHdL5yXvq2btcjl/Nzl9xMA7e/2dAu9+ezzz6lxvT2y3/JdE/a7nBf6rbNX9nNe+/o3qHE1Equ0GXgZ2kvKmH2lak2uL2fP2bBz3nOy/jjvJfF9ZQU1lY9vn9lXU2wzICDgHwbCwg8IaEE01dSPJxLo61s12fJ5bc7feKNoo//FX/yF6vunv/meqP2ud/561P7Bo1o3jU2otjbtBnBmXUdK+o4eParG9fVKQSDWUweA9/zGu6M268b/9NATatw3vvGNqM1ZhwDQQXSQNQdVVB/Rj+NGeOLcOTG5LW3JbsHgkNB0yZSJOCPK0UZK8menSHN/1mQ8Zum7FIva1GShj7l5iayztQo48nDXLq1PeOSZZ6I2f+fbXvkqNe6nj0uk3Yo5vqdMuD/5N/971N42oj+rQK7Q/LwWkc7k5Fy1JU25caLm+NxfovlIGZaWpku0U/0Dooyti+coO8/2vVCEJ35AQAsiLPyAgBZEU039Wq2G2bl0JDToAAAgAElEQVRVs3LnTm1GH35azDoPHfWUy8lu7Ec+IiIJ7/nNd6txZ6lsVrZDC2Bw6aYUJZt0mGSHDtLmY60/AJiicklsonZ36Xqhd99zb9S2rgTv1lcq+nsObpbot9O0S87RfgBQr8l34Yq4gC7pNDYhenP9pmrvlmFxQSantST1FopmHB8TZsOeKy4dduOB61XfM4cl0m6QklBsUhHv5J81CTw12jHfsn1H1P7aN76pxlG+FN76tgdU38CQJAilc/JZCaPJuExMSdlclwxFA8ZMaSw24R2Z31bOnGEr6bKMNr+vPaEFWNituKTK85qfdnmEJ35AQAsiLPyAgBZEWPgBAS2Ipvv4F7XHWSQC0JlYNrLpQx+S6tv/8l9+gN6jBSq2bRF/bsqUoK7TT9zYqAg8dHXreXD0nKXz2P/KUFaZzbZimo5FFgBgckaoLaaoAGBoaChqc9TdKRKaBIDt20eiNpfkAnQZZyXEYXxOVYbbZCiepQi9OJ04S0PFKCrxnPHPOaNwhUp0X6Rz5bWc4/mTujz1Tx6XfQLQKc516vluHZFzvGP3daqvVBV/vUSuuzfPPM7q884+D+W1LZOdIJ+fqcqVgq7XwPf0ej4+nzfrxzOFlzD3nI0QvRKu+MR3zqWccz91zj3pnDvinPuTxt93Oucec84dc8593jnXfqVjBQQEvDywEVO/BOBe7/0tAG4F8IBz7g4AHwPw5977vQDmALzv6k0zICDgpcRGaud5ABc5mLbGPw/gXgC/2fj7pwD8MYBPrHesbDaLO+9cFdV44gmta8aCEiwgAejot7/6q7+K2n/4B3+gxr3tzW+O2sNGU61EAhDVspjiWUNRFQsSObViqCcueVUhsy5pouf6hyQRpS2lj//8CTFnb9x/k+pbzot5WKRaAqmUdmlOnjwdteOmoupJMpdvue1WmW9FR7SxWdprqL5YTMxINvUX53Xk3tCAuCNTE9q12k1ReKOjQrM681lMPz79tK6Cy8br0Cb5rHhCU3H/+L95Z9SueP0s2zYkFCkFvqFW16ZxjfUUjfAJJ8dY3UGmnlXdBRO5x+a8Tb5hk54/24qKeErEuSSq72oIcTjn4o1KuZMAvgPgBIB57/3Fu+c8gK1rvT8gIODlhQ0tfO99zXt/K4BhAK8GsO9ywy73Xufcg865g865g/aXNCAg4NrgBdF53vt5AI8CuANAt3Puoo0yDODCGu95yHt/u/f+dltOKiAg4Nrgij6+c24AQMV7P++cSwO4H6sbe98H8E4AnwPwXgAPX+lYqVQK+/athnYeP/686jtNlFUypf1Wzh4bGBDK68IF/Vvz+S/+bdR+96//E9VXLotP29MjIbYlQ7swpXbhghaXYGHIdhKe5PLIAFAj/8tmtF1/nRhLNguss1OELWam5X0JkxGWJ//x3/7lX6q+3/qt34raY+R377v+BjUuQX78oSceV327d4xE7RPPiOhFxsyj3YmvunOr9vQGqC7A3Jx8l0OHnlTjzo8LrdhrdPX/xft/O2o/+l9+FLV/+5/9jhp36rRcJw57BoBjzz4bte9/k9RnqNd0WC5TbJwlCQCViswrkdDngPeOikW5LqbyuPLxLWIxWYa8N2ApOq4pcal5/cLovI3w+JsBfMqt7jTEAHzBe/9159wzAD7nnPs3AA4B+OsX9MkBAQHXDBvZ1X8KwG2X+ftJrPr7AQEBv2RoauReLOaQTq2aSrWqpjtKZTGZbHQXZ3QlO4Qe+3f/4dNq3Pv/xe9F7Sd+fkj1veIVQm0lKVPv9CktQrG0Iub3li1aH26aogGLRTmGi5sy1rSJ2ZnROnK+LrTL3JQ29cdHJarvtXfdHbW//nffUONyZEYvF7WYx51vkFLQhTzRkV6btpPnhWLr0KwRklWZ/65eoSpLeR391xUT9+nM6bOqr04m67Fnj0ftC9Naz657UL7LHa+7V/XtvuHmqL1lu2T/tbdperMtTmWmTY2D67YLrevJrYt3aJq1m6MBDdXHWXJxp5fM1IRcs2xGXLW5Oe3+dXfRNVvS55FdhLk5uR+doew4ItRD+xKxROO+2iCrF2L1AwJaEGHhBwS0IJpbLdd7lMurZmQ+r6PieJe/t0+XN+ol079OO50upqd/3xvfFLXPnDmj+r5KlVLf+Y9+LWrv2auTOuZnJJKMd/EBIAaJHltcofnHtH1VKooplzOmfmFFzLpdO3eqvolJMRu//W2Ric4Y+evf/d3fjdp2x5+Te44cloSjZFZHnE1PiduyyUid52dFwOOGkW1R+/RxnUSzOC3S2BzVCAD1uMyLI9Xe9MCvqnFVSohpS+rv2dYuJn25ItfdKknHKWrNlpZKUGSdo7JWXNHYztFGwfFru9POYioc4VczrAFH3VmtRWYA7PvUuHUe09H8N7i5H574AQEtiLDwAwJaEGHhBwS0IJosxFHH8sqq/9vVraO0Tp4Sn3x4h9Y8Z6ELzoC6MK5LXO3bL+W1Dh3SdB5HTn2T/Oe9u7WfPbxZIvcqNU2ZtJHGfD9lzM0ta1qOs/9cTftz6aR8l6zx3UkCHsdOnoja23fvUONGhkUvf2luWvX1UZmo3i7KGizpPIlN/bJvsmfbNtV34hnJkpueluOvmBLX//if/GbU/vLX/7Pqy/bJXkO6RzLyJqem1LjX3CX048yC3vcB0a7xds5u034w3x9WFKUtRaXIEnKCaya0jiPm1itjbX18vh/ZP7f1AxisnQ/o/QVF2ZnPWi8D7+Ln+Q06+eGJHxDQgggLPyCgBdHcarnVKiYaWu97TdLIwpKYeYcPH1Z9qtIomWGj53Wl2M4uod/+6EMfUX3/33/4VNQe2S6m7Ve//CU17g13CpW4dcuQ6quWxVxmjcDNg3ocJ3zMzZhItS6h95bmTFXWtHzPN79RotiyPVpjbnxU9O0GN2lNv4QTczNNyU4/+cl/UeN2kVDJ+bOn9PHHRHOvZ4/QnZOzc2rc01TianhEuyOZbjHvB0lQ4p43v0WNe/xJSdqpzRpKcA2KrVTV7lOMhCzakvqW5pJlXNPAmuJssq9H51m6jaNK2Uy3lB3DJgHx8XleVrBjrffw+zaqvRee+AEBLYiw8AMCWhBh4QcEtCCa6uNXKhWMja+GxN54sxGaJF/pxCkdbsv69umM0GgrRa1PPrRFaK7ewc2q7743iRDnV770xajd2aszARcpA+35v/+h6hsbFcGH++69J2qPjmpN+QTxcuWCzkLkUsfzs5oGLNeJ6qKMsOE2XWq7QDr16ZQWwFhZkmO2kRvY2633CbYTJZgyKV233Xhj1J6elDm9Y99+NS5PmvWuov3R4Z3i8x87LeenYkNlySetWSqK/Ng4+fHVFX0MrmfnTG079uv56NYHZ4rN+s/sN9u9gcXFxcv2rRd6a2scMH3Ix7D+Ovv8ti/6vBCyGxAQsBbCwg8IaEE01dSvex9RMRmjRT83J1RRqazNKc7Cm55nbXtTzjgrVNnMrKbKllbEvPqjDwvV958e/qoa9x0SvbAllx9/7KdR+1vflei/1/zKK9S4yTHRAhwbnVR9M7PinvR1mVLeFBmYSEk23Vvf8VY1jt2TS+glcgNmZiR7bscOTbcdPHhQ5pHTbsAt+8Wkn5wRvbx4Wl+ze94k56dsxCt+Rhr5bZRBmDWl0zjzMBbXUYhswifoVq0bezbWRs8vkykZS1AJMDKjTSVsRbFZGo1f2/JXrLe4UVPfqk2zO8Kwx1jvmC8wOS888QMCWhFh4QcEtCCaauo7F0O8IU/c2dWj+lIzYupPGUlqTzunvCvc26V35B3t/CY7tC7bMWIKXtMo4wUA8aQ2tz/2f/1Z1P7j/+V/Un1Zqs7b2SsRc9Nzene+WBWDK5XVQhz9MXE5tmzSO/LzC2I29m+SaMByVZuei8uyA10xySaOZLM7SEfuxj171Lijh8UU37pdJypto2i9RSrl9eZ3/Joad+qsRPhlu/W1yJKE+SnSpZtb1Fp0F0zpLQbLTifIZLe77mwqWxnrBPXxmVovcs+a+spFMGzAei7CWrCfbctyrXU8NvUvld7e0EdHCE/8gIAWRFj4AQEtiLDwAwJaEE318QEf+Sadndr37SSf8PyYzroDZXdxVtzgJp0Vl2gTX+mU0XlPERW1uMLlqDVF0kY+/wc+qMtwP0PZaCXSb0+0aZ9zZlIEQp54XJcDL9ckku+5kzpCkUXR/9nvSY2AtKHbODuN/XgASGXldYr2ApYNRVryck57hnSUY1c/lb+elay+Z57XZc+u3y+69ytFHaHIwqEDlL04Na33b3p6JYtvfslo85OPy/6+9fE5qo8FOwDtP9fIEa7n9XzZd1+PzrOU2nrCGWthvTLZjBcSufdCseEnfqNU9iHn3Ncbr3c65x5zzh1zzn3eOXd5MjIgIOBlhxdi6n8AwFF6/TEAf+693wtgDsD7XsqJBQQEXD1syNR3zg0DeCuA/w3AH7hVW+teABdF1z4F4I8BfGK948Ri8Uirnk12AMhRRJelZJiu6e0XGm1wkzZR420yzurlcaQgm1asQw8AJ6ny6lbjSmTJrO7qES36sUmt/bfngJQa/O3f+eeq71t/9+2o3Wa03XaO7JZ5nCMNwr2a7qnS2yYMlVh1cu4WqLzW9qymTz/28f8nan/ps59XfWVyA1Kd8r5tu3ercUWipZYKOhot0yXnZ4IiCCemteZekrQLWfcO0NcpTvr71jROULSe7WPzuJ0iCG203Nmz4mbYqFK+N62IxvHjUh5s/34Rl1k/ym5tvb8OQ0MzWPvPzj865EucpPNxAH8EoUL7AMx77y9e+fMAtl7ujQEBAS8/XHHhO+feBmDSe/8E//kyQy/7W+Oce9A5d9A5d7BklF4DAgKuDTZi6r8WwNudc28BkALQiVULoNs5l2g89YcBXLjcm733DwF4CAB6+/pf3FZkQEDAS4IrLnzv/UcAfAQAnHN3A/jvvffvcc79LYB3AvgcgPcCePhKx6rVapFwgQ197CH/2WaSMRU3MKRLVzPYP7K+GPvn+YJkWCVMuGSRs6/i+vTkiCorVWVculP7zx1dIhwyOqkFKjty4vvmMro237adUk9glLLi8mV9rrr7hQJbKervuZCXec0sSt/citZyL9bIL+7QPm0H1TzIkghKzJQDXyTxCiuiwaHVSco6LBR0dltbu9CnVfM92U+uk8a+N/s3dTJAvQmHXUv00v6ds+5sSC3Pw76P+5hmjJkS13w/LixoUVH+bB5n97qYwrM+frRd1IQy2R/C6kbfcaz6/H/9Io4VEBDQRLygAB7v/aMAHm20TwJ49Us/pYCAgKuNJpfQqmJhvmHCOm2uZUn//LrrdOnqdjIVOeJvYsoIN5B5ZU39kZGRqM36fpa6WWENNUPJZEkTvzQnpniiXbsLPQNCA05Oavqqi0p+F4z5vUIm3zxlsdVs2aaS0D+WRquSrZfKCg3VYdyRJJfvjuv5F0ilopNKaNdjmn6cp3OVSmuBjQpFCqZSYs6PT2vXh8VTrBnNpr8nUQ5Lh8Xo9Fh3YT0TXs2XPttScesJcfDr9cpk27JcDDbp2V2wWXvrRe5dnP5GA/pCrH5AQAsiLPyAgBZEczX36vXIBE8Z83iZEkoGaNcaAG1ZAjU2d8zuLksdW1N/YEAi/i6cF7nnTE7vrE+OiTCE1UZjk6xIpmwmpxOOOBrt/KhOODqwT6K7lhZ01B3Pf2lJdn5tlCObgPVlvUO8VvLGoqlEe+KEVOP96cEnVN/Sgpjjw1vENSkaMzpP8ubtaX0e2QTOpaRvbnpGjds2LAxOpaQj92qVy5eTckbfjx9f1pyv0DzKZEa3J/V81yrXZV9bU58FPNYrocWv7Y4/v+YowfWq5dp5XK3IvYCAgH9ACAs/IKAFERZ+QEALoqk+vvd1KjWt/fPJSfGtc0akg4mRxWWhuawvNjMlGva2TBH756zhP7xtixrHvpjdJ0inLi850G/2JJYWRdP/zOmTqu/Vt94StQf6elVfjsqDpSjLLGHOVawufmxhyQh90vmJ0fuyHVpUdHirZDbefOBG1XfrzQei9tGjh6L20oreJ9ACFdq5LJIwB+uI8LkHgDaiCOtG7N5z3QTaz3HGj1W+sPH/eZ/A0+3i2y0dtragxlolri73ei2ouhFbdEYo++u8n2OPnUjI/WEpzYuMaVEzxGsiPPEDAloQYeEHBLQgmqy5J1gwJl+RotjmZnVEnqffpyKZZBmjWc/JD9ZcKxTEbOfSUmzyAkC9LuYlV1AFgFRS3AU2nWsVTUPNUl2AWlXTLtUKRXpVNV2YX5q/bJ8157NdYvJVy3qO1ZJ8z2pR3J2FeX1O50gQY2FOU2x8/tlEZcoVABJxcX2sa6U17OTvxbw+H0xlXRIxxzry1GXN3Dhr6ZkklTq5RZzMsx4tVzfuAs/RfjbfZ+xOXiIWsoauHqBdCT6+jfbj83MpJXjxu22MzwtP/ICAFkRY+AEBLYiw8AMCWhBN9fHjsRiyHat0xcSEDmXt7hF/nUsPA7pMdo6yxdarY8bUGAAU8+LDrSwJLWUFHtnH4iw+AMhl5ZgXRUMBoFDStN/0lHy3DiOmMDMhQkVMvQFAT1aonKF+4cDaYtqvnJ8W2nLB7IfUaQ8hm5bQ3p6czkLs6pTXPd06s45fs+9rQ5hZIGVpUZ+DtWiuS7LWqGYCU2+ApvPqTMvZvQDya2OG4vVxzmiTvvWEOPh+A9b38deqZ2f3mDhD0d5XfIzRUalHaMU2OKTbinSUy77xudgQwhM/IKAFERZ+QEALoqmmfiwWj7KPrO7Yrl1SxvkcmTsAEI+L/dJN2UsXJrSefbkkZmlXp86+8kTrVIgqqxlTn020gtGz0/rnYrqlUvqznqXIvXLBlIUelczA4U1aP3D//utl/l1yTJtBOLso586Z+TOFx9+FowkBYJleFwv6eyqqi6itS2g00rov5PUxmH+LO6KhoM35GJiyM9l5NRLHqJcu+/fGQWiOhgLjbE6abwxr03JxExpIcn/qPgI07VqnedUtXZgXM/1CQdOzcfJOinm6tmaOy2Tqp5NGV78x5Y2q2YYnfkBACyIs/ICAFkRTTf22tjYMbV5NijlzTpvz20dEWrqnR+vDsRlWq4o5ODWhpfznZmS3+4YbblB9eRKscGSGciQdAGTIhF+c19GF8W3DMic2Q4vGrCM9voEeXc12cV6i5Eqmb3paduh5B31uQZvpK2TOx4xtl03KTvvUlETnJWPaNKyUaJfchLulkpQMQrvpNVMQxdGufrupGJyg3fQSmbbFFe3iFYvy3XI5PcfpablHEgnRMfR1beqzKIoVLSkWZR4s+Z3K6GfePEU2HjhwQPVxRGUxr69FjOZSI3bnhz/4rho3sl0ER0omk6anS9zXUySQsmu3lpmfHBc3cdaUTrv4tfMbyxkKT/yAgFZEWPgBAS2IsPADAloQzY3cS8QjnXbOYLOvrY/PWUrz8zJuflZnlVXIB3XeRFhxBh1l4LW321Mg77P0FUexpbsl0rBWNuIPFIm1XobVqTOnVV8bUzQUPdbbqwU72ttkH6JS1lFsHDHHeyPVqh0nr8tGRJP3GjjaLb+iffxkWl6vJ1DBiWTtKR1xxtlzExPjqi9OPFp3r9wTBbPXsEJRmTFT9qxUle82Py/+eUenyYykTMzJyUnVV+2TaNGqEQRdIXES3vcZHNTiLPMLct9aKm6aMiWTabm2Njs0TXUS7Pm2+qNXwoYWvnPuNIAlrIrhVL33tzvnegF8HsAIgNMAft17P7fWMQICAl4+eCGm/j3e+1u997c3Xn8YwCPe+70AHmm8DggI+CXAizH13wHg7kb7U1itqfeh9d4QjyciM358XJt1Fy4INWdFBlhHnhMVJkzkHmvwWTPdJv6sNY6TH6w5xUlAGUqomV/USRdok/l29g2oLnYXrLtToqlwZNncoqZuOMlj1JxHNntrZP+t5LVpWyZ9O65YCwCO3JPeQaHRak7fLsVK/bJtAFhYks/rrxLVl9Bmbp5LbRlhlUWa8+lzcn+cPXtWjZuZWdvQ5OtbKAvtGjfz6O6R67R1eEQfhMq91dr0PbF3n+gVbt2xM2qvmPstRnRqwbhnP3j0B9JHgjSVkhY34aSdzZt11GdH56o7spC//H1usdEnvgfwbefcE865Bxt/G/LejwFA4//BNd8dEBDwssJGn/iv9d5fcM4NAviOc+7ZjX5A44fiQQDo7Oq6wuiAgIBmYENPfO/9hcb/kwC+gtXy2BPOuc0A0Ph/co33PuS9v917f3tHR8flhgQEBDQZV3ziO+cyAGLe+6VG+00A/lcAXwPwXgAfbfz/8JWOFYu5SJDACjIwhdRpdPU5DJP3Bqzu/eCgeBtWaIH3EJhiszrv/Nnnzp1TfZy1Nr8gvtT5cb3XEKeQ10yXpuLyZaFuVkraX3RE052nDMVnn31GjRseltDhCxP695b3Q6pE8TCVBei9Em8EKhO0R3F+XI6fSOkf7p0kmJ/r7lN92S6hwPh7LS5pv/XZ46fog3U9xedPnonaqQtyjmOGIuWS35dkENLYzWRxTs1oAROuC3jkqDZoz5yROXZmtWjJ1LTM66677pL5mofcyeMSirt1q67lMLxN9gY6UnIO7P3NIqjpjF4jk8+tHr+ywZDdjZj6QwC+0tg4SwD4j977bzrnHgfwBefc+wCcBfCujX1kQEDAtcYVF773/iSAWy7z9xkA912NSQUEBFxdNLdMtvcoVRoUjUkrWyYaYmFJm6XRewDMzIm5U4d2FzpJK65Y1iblxJS4CENDQlGNmQy/ffv2Re2aN9FolIk1SaZiIqUzwnJ9Qgkmc3pDs70g7kLJlPmeIy3A0+fOR+1zF7Q+4fadksnozTZNmVycPFGHywVNOc7TOZ6ZmVJ9Y5NivmZp/sWSdp9OnDwdtReMfuApMm3Z9ekwJcV3kgDLj3/yOdXHrtsmEi3pH9QUaUdGhEqsLiBTsiUy5wsmAu/EKTHnb7tFP+ceeOCBqL19eJvq++RDn4jaZ8/KNeOScACQTcs5mDalwheXxaQ/RfOwunoTY3Kv3nGHdhcyDU3FSkXf92shxOoHBLQgwsIPCGhBhIUfENCCaKqPX61WI9rO0i4chjo2pn1a1iRnWsrWI+Nj2swmfs2+06gR9ty2TXw4q42u6Dzyx9NZ7cfnyc+cX7Shm1QrzmRUKWqRwo/vueceNS6ZEIqqrU1TWyyquUQ194b6dcYjZy8uLWpVnPlZ8flZHHQHqSQBmvq0WYiLFKq8Hn06NSX+rlXPuf56ER/dtYv3NTT/yH68jRXhPg7pXpjT4dI1CrFdNueDtTcX5vX78uSfz07Jvs+hg0+ocTfu3x+1SyYUN5mQ+5FDurs79X3F97e9923J+CshPPEDAloQYeEHBLQgmkvn1WqRSc8RZoDW2ecoPgDIZETUkU03eww2k2xkIJtCfAxrej7//PNR25bX4my6RcqiWjDZeVzSKWEyDdnEzhvhyfk5+T6lFTHTd257pRp36sTxqN2uLWw4EoPgktkxQ03GaiRaUtcUWBdlHmbSkhHW3aWj1lj33er254neK1CUYMW4YDkSnlg0wiqLC3Jt2MSumkxANoHbU/qe4Iy2NhKwL67ojMeePnGF4uZcLZNA6pGfaxM+lZBjdtN5K5tsyCLR1efP6+zCwX5xi/i8wdCW7eTWVcr6+CIuE8pkBwQErIGw8AMCWhBNNfWdc9GOOu/UA1oowyYnMHjXdr1sPyu8wbvpHN3F7gEAHD8uZrTVuuNdfU4CGtqyVY3j3W7rjuRpp73dRC+208/wIkUojhvTcJn12xL6GJ0ZMW1XFqh8lKnoW1gSMzpW03UBuFLvQTK368a8rJCOYd7shFeK4v6wy1ErGx25NvnS/aZqb60kYwtk6qczelycTOCVZT2PhYLMuT1JVZeN1mJXlq6TqbUwQPqKB8c0C9ROW/5J+i7dnfr+TpNPVjd1GFy9TOPkGOxGAMBAjyQ+tZkyX+nGulpw+thrITzxAwJaEGHhBwS0IMLCDwhoQTTVxy+Xyxht+KtWbINr2DGNA5jy1ySmwHXuACBJfpull5gOeu2dd0Ttnx/S9My2PZIt1mWkwljck2v4Lc1p+rE3K/Oq5rXPxf503tTmc6zfTvXmyob2SzquJagz5mpU2rszJf5+ZUXveWToXJWWNLW1Mif01VCvzGlkq5ZVPHr4SNTOGb38GSqHXaT9hOEhLdgByiZrd5qCTdPdWViU61c034X3bOJt+paen5XPvvnmm6N2PqP3Xoq9sm8wa7Lnikvy2f1del8pEaP6gfSde8zxQfsjKUPBeqoT0E50XGFJX/c41XzsaNfnGxfvAx/ovICAgDUQFn5AQAui6WWyNw2smouTRvNs7LzQJCwEAQCO9OE7idK4Ye91atzmzZuj9sykFpdYzosJPD4qggZW235xTlyEgQEdOVUjIQcWRYjXtQkcGxbRiNMnTqq+kZ3bo3Z+QX/23JSYkV0ZiQJjugfQUWdzU/pcVchl6iNXpTOrTdRZOsc1I95QXJbjP3v4yaid7dBJNCvLQrf19Gnqs58osE5yyVYypow10W82Yq5C86hRSTGX0LaySl6p6FuaacaZSblmuQ5titeoZHlX2tDE5NZt2zykujgyMBET+u363bvXHLdlUN8vKYqOZJrYRp8y4uaRXW9EbNrEsrUQnvgBAS2IsPADAloQYeEHBLQgmurje+9RaWS8sU8PAFNUmtiWIi5SWC1n1q1s1oKDJaobN2d89ySFCHNfxYRPzs4IlbOnvkf1tZH4QZzE6H1Fz7dKIcfHnz2q+nZul/DemBEjKdM+RG+n7GUUl3S4bapNfMK0CX3OZYVe6iEhh7TxaRfnxfcdGtA+Z5Ey6G6+SWrDZdLGLyZ/tGAowWpRrkWZykL3dmuKNEY+6QESOgV02fAK1dizvi8LfdjsvDiJdnRTlufZc6fUuLg4/fEAAAmPSURBVALRoAmnqbLDT8s+B/vqgA4FZzEZvkaAzg7NZrOqb/yCnG8WI7Fimyw+arNKF+dWz4/Rb10T4YkfENCCCAs/IKAF0XQhjotlgIsrOksr7uQ3qCOpzddygUUjxDQsGzN9mspJTYxpXXPW0ufIrITTv31zM+IGLMxqcyoRE5NyK1GHdaPhP05U39KCNoF5ztu2Dqs+jhQcGhIqsXBJtqLYcwODulwyz7FE86qUtQ1YJcG/7Ts09cQma52s6ulp7T4xjcaZi/YY81yuy9TrOnFCTG6ruefou7CJbfX9WKilYmzdEt07TN2mTDkwPr7N+iyRK2ejOfkcbBqSe8m6BEzT2ajVGXIv2XVImFLeSaof0BPTfTccWD3fh49pF3otbOiJ75zrds590Tn3rHPuqHPuNc65Xufcd5xzxxr/91z5SAEBAS8HbNTU/wsA3/Te34DVclpHAXwYwCPe+70AHmm8DggI+CWAu1Kkj3OuE8CTAHZ5Guycew7A3d77sUaZ7Ee999evdRwAyOWy/vbbbgKgq94Ca4tcAFoso7+/P2rv3btXjbNm2Fp9uZwkZHDiDaBNVI4EBLREN+sgZNJ693WetOMmJnQE4eCgzJ8r+AJaOpxNvgVT6ZbnHzOaft3d3fRKJtnXrQ0y1hMcGRlRfVwBNubEGywa9oLFVJhtAfTOu9IgjJtzRd/Nmsd12pHv6ZH523uW51Gq6B3/ZXIzeFyhoN0nTwkw1sRmURcf064KVx3m+4XvI0Df03ZXn3f8+bpYbcFFinLsMVWYL5aMO3joGSwtrVxRa3sjT/xdAKYA/Dvn3CHn3F81ymUPee/HAKDx/+B6BwkICHj5YCMLPwHgFQA+4b2/DcAKXoBZ75x70Dl30Dl30D7JAwICrg02svDPAzjvvX+s8fqLWP0hmGiY+Gj8P3m5N3vvH/Le3+69v90GJAQEBFwbXJHO896PO+fOOeeu994/B+A+AM80/r0XwEcb/z98pWN1pNO49cZVH790/Q2qr0CliMan9G8IZ9rVyXuZHtP++bkzIkrZmdF+VBtFdOU6hP7pyOlxPVmiWural0xShhjTOAsLWhiiRL7Z3r1626NMwhkjuzKqj31JppSsYCf/gFpRUfYfucxSIq791tNnJGtwak4fY7kg80+n6YQ7fbtUiZqrmWdIjfx63ofwZh6JlJwDG3TGoqvJjPjg1sfP9ch3HtpqsgSplBfvD7Un9UOI/Wxb3s0RfWhrLfAeRV+fiIxYepP3XpZNbQGmePn+4P0DADh7Vu7vukncuzC2WqI7kTiGjWCjPP5/B+Azzrl2ACcB/A5WrYUvOOfeB+AsgHdt8FgBAQHXGBta+N77nwO4/TJd97200wkICGgGrkjnvZTo7e7y97/utQCAXJeOXuLIvUpNU0NJSnjgxA0jLQ5H6gRxE5EHomE8RXc5o2jAx5wxFVW5b6Ug5lqnqWoaI903I52PeUpmsRVOmbZk05wj0wCgb0BMVpuwwokcTGH29WlRkTxp1vvq2kkvU5NCu/YZsQ2OQLNzLJCOHGviTU1qARamvZbyOgKSTdsYzclSsByFZ6vI8jlg94nvI0BHDVqKlPuYSgWACp1/jsDbs0cneLHpv1Iw9QkoKW1gSK6fdeO4zkOmQ8/j/Ojqufr4n/4pzp09+5LQeQEBAf/AEBZ+QEALIiz8gIAWRFOz8+LxRBR6aWvnsa9XLetAnyzTbynxt2wWFfu31hdjf5T9QEu7sN9txQ62bhURDU4yGz2vw4/jpHn+9NNP6+NTyGrVcDJJ8lXzVGZ52fi+pXGhO22m13Je/MW5BcoSLGgairMVl4pGRKMqNFqG6M1sTn8Ws50Vs09AyX/IdQmVFW/T153vgw5TE499+X4qGW1rK6aJErShsky/cd/ho8+ocd4RZWe+ywLtS1Sh3WfeY/neoz+I2kkj2BmjMGAbWs5hxgskuvLtbz+ixu0joZJdu3apPndxT8hd0b1fnc+GRgUEBPyDQlj4AQEtiKbSec65KQBnAPQDmL7C8KuNl8McgDAPizAPjRc6jx3e+4ErDWrqwo8+1LmD3vvLBQS11BzCPMI8rtU8gqkfENCCCAs/IKAFca0W/kPX6HMZL4c5AGEeFmEeGldlHtfExw8ICLi2CKZ+QEALoqkL3zn3gHPuOefccedc01R5nXN/45ybdM4dpr81XR7cObfNOff9hkT5EefcB67FXJxzKefcT51zTzbm8SeNv+90zj3WmMfnG/oLVx3OuXhDz/Hr12oezrnTzrmnnXM/d84dbPztWtwjTZGyb9rCd87FAfy/AN4MYD+Adzvn9jfp4/89gAfM366FPHgVwB967/cBuAPA7zfOQbPnUgJwr/f+FgC3AnjAOXcHgI8B+PPGPOYAvO8qz+MiPoBVyfaLuFbzuMd7fyvRZ9fiHmmOlL33vin/ALwGwLfo9UcAfKSJnz8C4DC9fg7A5kZ7M4DnmjUXmsPDAN54LecCoAPAzwD8ClYDRRKXu15X8fOHGzfzvQC+DsBdo3mcBtBv/tbU6wKgE8ApNPberuY8mmnqbwVwjl6fb/ztWuGayoM750YA3AbgsWsxl4Z5/XOsiqR+B8AJAPPe+4sqKM26Ph8H8EcQyb2+azQPD+DbzrknnHMPNv7W7OvSNCn7Zi78y6UNtSSl4JzLAvgSgA967xevNP5qwHtf897fitUn7qsB7LvcsKs5B+fc2wBMeu+f4D83ex4NvNZ7/wqsuqK/75x7QxM+0+JFSdm/EDRz4Z8HsI1eDwO4sMbYZmBD8uAvNZxzbVhd9J/x3n/5Ws4FALz38wAexeqeQ7dzkZRuM67PawG83Tl3GsDnsGruf/wazAPe+wuN/ycBfAWrP4bNvi4vSsr+haCZC/9xAHsbO7btAH4DwNea+PkWX8OqLDiwQXnwFwu3muz/1wCOeu//7FrNxTk34JzrbrTTAO7H6ibS9wG8s1nz8N5/xHs/7L0fwer98D3v/XuaPQ/nXMY5l7vYBvAmAIfR5OvivR8HcM45d1GT/aKU/Us/j6u9aWI2Kd4C4Hms+pP/QxM/97MAxgBUsPqr+j6s+pKPADjW+L+3CfN4HVbN1qcA/Lzx7y3NnguAmwEcaszjMID/ufH3XQB+CuA4gL8FkGziNbobwNevxTwan/dk49+Ri/fmNbpHbgVwsHFtvgqg52rMI0TuBQS0IELkXkBACyIs/ICAFkRY+AEBLYiw8AMCWhBh4QcEtCDCwg8IaEGEhR8Q0IIICz8goAXx/wP7n7OQltSPqwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Displaying an example:\n",
    "i = 0\n",
    "class_id = class_ids[labels[i]]\n",
    "readable_label = class_readable_labels[class_id]\n",
    "print(readable_label)\n",
    "plt.imshow(images[i])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw through the previous notebooks, this `tf.data.Dataset` instances can be simply passed to Keras models for their training.\n",
    "\n",
    "\n",
    "### Wrapping Up for Estimators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to pass our dataset to an Estimator, we can wrap the iterable inputs (`images` here) into a dictionary in order to name the content.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch = {'image': images, 'label': labels}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We know have our input pipeline ready. We will reuse these variables in the next notebooks. For clarity, we wrap their definition into easy-to-call functions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _input_fn(image_files, image_labels,\n",
    "              shuffle=True, batch_size=32, num_epochs=None,\n",
    "              augmentation_fn=None, wrap_for_estimator=True, resize_to=None):\n",
    "    \"\"\"\n",
    "    Prepares and returns the iterators for a dataset.\n",
    "    :param image_files:         List of image files\n",
    "    :param image_labels:        List of image labels\n",
    "    :param shuffle:             Flag to shuffle the dataset (if True)\n",
    "    :param batch_size:          Batch size\n",
    "    :param num_epochs:          Number of epochs (to repeat the iteration - infinite if None)\n",
    "    :param augmentation_fn:     opt. Augmentation function\n",
    "    :param wrap_for_estimator:  Flag to wrap the inputs to be passed for Estimators\n",
    "    :param resize_to:           (opt) Dimensions (h x w) to resize the images to\n",
    "    :return:                    Iterable batched images and labels\n",
    "    \"\"\"\n",
    "\n",
    "    # Converting to TF dataset:\n",
    "    image_files = tf.constant(image_files)\n",
    "    image_labels = tf.constant(image_labels)\n",
    "    dataset = tf.data.Dataset.from_tensor_slices((image_files, image_labels))\n",
    "    if shuffle:\n",
    "        dataset = dataset.shuffle(buffer_size=50000)\n",
    "    # Adding parsing operation, to open and decode images:\n",
    "    if resize_to is None:\n",
    "        parse_fn = _parse_function\n",
    "    else:\n",
    "        # We specify to which dimensions to resize the images, if requested:\n",
    "        parse_fn = partial(_parse_function, size=resize_to)\n",
    "    dataset = dataset.map(parse_fn, num_parallel_calls=4)\n",
    "    # Opt. adding some further transformations:\n",
    "    if augmentation_fn is not None:\n",
    "        dataset.map(augmentation_fn, num_parallel_calls=4)\n",
    "    # Further preparing for iterating on:\n",
    "    dataset = dataset.batch(batch_size)\n",
    "    dataset = dataset.repeat(num_epochs)\n",
    "    dataset = dataset.prefetch(1)\n",
    "    if wrap_for_estimator:\n",
    "        dataset = dataset.map(lambda img, label: {'image': img, 'label': label})\n",
    "    return dataset\n",
    "\n",
    "\n",
    "def tiny_imagenet(phase='train', shuffle=True, batch_size=32, num_epochs=None,\n",
    "                  augmentation_fn=_training_augmentation_fn, wrap_for_estimator=True,\n",
    "                  root_folder=ROOT_FOLDER, resize_to=None):\n",
    "    \"\"\"\n",
    "    Instantiate a Tiny-Image training or validation dataset, which can be passed to any model.\n",
    "    :param phase:               Phase ('train' or 'val')\n",
    "    :param shuffle:             Flag to shuffle the dataset (if True)\n",
    "    :param batch_size:          Batch size\n",
    "    :param num_epochs:          Number of epochs (to repeat the iteration - infinite if None)\n",
    "    :param augmentation_fn:     opt. Augmentation function\n",
    "    :param wrap_for_estimator:  Flag to wrap the inputs to be passed for Estimators\n",
    "    :param root_folder:         Dataset root folder\n",
    "    :param resize_to:           (opt) Dimensions (h x w) to resize the images to\n",
    "    :return:                    Dataset pipeline, IDs List, Dictionary to read labels\n",
    "    \"\"\"\n",
    "\n",
    "    ids_file = os.path.join(root_folder, IMAGENET_IDS_FILE_BASENAME)\n",
    "    words_file = os.path.join(root_folder, IMAGENET_WORDS_FILE_BASENAME)\n",
    "    class_ids, class_readable_labels = _get_class_information(ids_file, words_file)\n",
    "    if phase == 'train':\n",
    "        image_files, image_labels = _get_train_image_files_and_labels(root_folder, class_ids)\n",
    "    elif phase == 'val':\n",
    "        image_files, image_labels = _get_val_image_files_and_labels(root_folder, class_ids)\n",
    "    else:\n",
    "        raise ValueError(\"Unknown phase ('train' or 'val' only)\")\n",
    "\n",
    "    dataset = _input_fn(image_files, image_labels,\n",
    "                               shuffle, batch_size, num_epochs, augmentation_fn,\n",
    "                               wrap_for_estimator, resize_to)\n",
    "\n",
    "    return dataset, class_ids, class_readable_labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ImageNet\n",
    "\n",
    "For our more ambitious readers, the same process can be followed with the original *ImageNet* dataset, after acquiring it from its website ([http://image-net.org](http://image-net.org)).\n",
    "\n",
    "However, TensorFlow developers have made public the `tensorflow-datasets` package ([https://github.com/tensorflow/datasets](https://github.com/tensorflow/datasets)), which greatly simplifies the download and usage of many standard datasets (it is still up to the users to make sure they have the proper authorizations / they respect the terms of use for the datasets they download this way).\n",
    "\n",
    "We will not extend further in this notebook, as `tensorflow-datasets` has been already properly introduced in a previous [notebook](./ch4_nb1_implement_resnet_from_scratch.ipynb). The explanations shared there can be directly applied to the _ImageNet_ version provided by these package ([details](https://github.com/tensorflow/datasets/blob/master/docs/datasets.md#imagenet2012)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tfds.core.DatasetInfo(\n",
      "    name='imagenet2012',\n",
      "    version=2.0.1,\n",
      "    description='ILSVRC 2012, aka ImageNet is an image dataset organized according to the\n",
      "WordNet hierarchy. Each meaningful concept in WordNet, possibly described by\n",
      "multiple words or word phrases, is called a \"synonym set\" or \"synset\". There are\n",
      "more than 100,000 synsets in WordNet, majority of them are nouns (80,000+). In\n",
      "ImageNet, we aim to provide on average 1000 images to illustrate each synset.\n",
      "Images of each concept are quality-controlled and human-annotated. In its\n",
      "completion, we hope ImageNet will offer tens of millions of cleanly sorted\n",
      "images for most of the concepts in the WordNet hierarchy.\n",
      "',\n",
      "    urls=['http://image-net.org/'],\n",
      "    features=FeaturesDict({\n",
      "        'file_name': Text(shape=(), dtype=tf.string, encoder=None),\n",
      "        'image': Image(shape=(None, None, 3), dtype=tf.uint8),\n",
      "        'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=1000)\n",
      "    },\n",
      "    total_num_examples=1331167,\n",
      "    splits={\n",
      "        'train': <tfds.core.SplitInfo num_examples=1281167>,\n",
      "        'validation': <tfds.core.SplitInfo num_examples=50000>\n",
      "    },\n",
      "    supervised_keys=('image', 'label'),\n",
      "    citation='\"\"\"\n",
      "        @article{ILSVRC15,\n",
      "        Author = {Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and Alexander C. Berg and Li Fei-Fei},\n",
      "        Title = {{ImageNet Large Scale Visual Recognition Challenge}},\n",
      "        Year = {2015},\n",
      "        journal   = {International Journal of Computer Vision (IJCV)},\n",
      "        doi = {10.1007/s11263-015-0816-y},\n",
      "        volume={115},\n",
      "        number={3},\n",
      "        pages={211-252}\n",
      "        }\n",
      "        \n",
      "    \"\"\"',\n",
      "    redistribution_info=,\n",
      ")\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# !pip install tensorflow-datasets # Uncomment to install the module\n",
    "import tensorflow_datasets as tfds\n",
    "\n",
    "imagenet_builder = tfds.builder(\"imagenet2012\")\n",
    "print(imagenet_builder.info)\n",
    "\n",
    "# Uncommment to download and get started (check terms of use!):\n",
    "# imagenet_builder.download_and_prepare()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"ref\"></a>\n",
    "#### References\n",
    "\n",
    "1. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L., 2014. ImageNet Large Scale Visual Recognition Challenge. arXiv:1409.0575 [cs]."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}