Image_classification_with_Vision_Transformer_Keras.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/itc202/praktik/blob/main/Image_classification_with_Vision_Transformer_Keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j8cHKuVpbPrR"
      },
      "source": [
        "# Image classification with Vision Transformer\n",
        "\n",
        "**Author:** [Khalid Salama](https://www.linkedin.com/in/khalid-salama-24403144/)<br>\n",
        "**Date created:** 2021/01/18<br>\n",
        "**Last modified:** 2021/01/18<br>\n",
        "**Description:** Implementing the Vision Transformer (ViT) model for image classification."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U_PerjisbPrW"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "This example implements the [Vision Transformer (ViT)](https://arxiv.org/abs/2010.11929)\n",
        "model by Alexey Dosovitskiy et al. for image classification,\n",
        "and demonstrates it on the CIFAR-100 dataset.\n",
        "The ViT model applies the Transformer architecture with self-attention to sequences of\n",
        "image patches, without using convolution layers.\n",
        "\n",
        "This example requires TensorFlow 2.4 or higher, as well as\n",
        "[TensorFlow Addons](https://www.tensorflow.org/addons/overview),\n",
        "which can be installed using the following command:\n",
        "\n",
        "```python\n",
        "pip install -U tensorflow-addons\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "pip install -U tensorflow-addons"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0AagSRX9S3kA",
        "outputId": "1f4ae01e-d86f-4ab7-e7d7-fecac367fb89"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Collecting tensorflow-addons\n",
            "  Downloading tensorflow_addons-0.22.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (612 kB)\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m612.3/612.3 kB\u001b[0m \u001b[31m7.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hRequirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow-addons) (23.2)\n",
            "Collecting typeguard<3.0.0,>=2.7 (from tensorflow-addons)\n",
            "  Downloading typeguard-2.13.3-py3-none-any.whl (17 kB)\n",
            "Installing collected packages: typeguard, tensorflow-addons\n",
            "Successfully installed tensorflow-addons-0.22.0 typeguard-2.13.3\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vdet8nRobPrX"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cub53zFfbPrr",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "b6bcfc45-8161-4cc9-d5b5-907ed0600e4a"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.10/dist-packages/tensorflow_addons/utils/tfa_eol_msg.py:23: UserWarning: \n",
            "\n",
            "TensorFlow Addons (TFA) has ended development and introduction of new features.\n",
            "TFA has entered a minimal maintenance and release mode until a planned end of life in May 2024.\n",
            "Please modify downstream libraries to take dependencies from other repositories in our TensorFlow community (e.g. Keras, Keras-CV, and Keras-NLP). \n",
            "\n",
            "For more information see: https://github.com/tensorflow/addons/issues/2807 \n",
            "\n",
            "  warnings.warn(\n"
          ]
        }
      ],
      "source": [
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers\n",
        "import tensorflow_addons as tfa"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VRSdY0gCbPr8"
      },
      "source": [
        "## Prepare the data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JycKsxf2bPr9",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "f5e11eaf-41c3-46bf-bb12-4532a961dc1d"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz\n",
            "169001437/169001437 [==============================] - 6s 0us/step\n",
            "x_train shape: (50000, 32, 32, 3) - y_train shape: (50000, 1)\n",
            "x_test shape: (10000, 32, 32, 3) - y_test shape: (10000, 1)\n"
          ]
        }
      ],
      "source": [
        "num_classes = 100\n",
        "input_shape = (32, 32, 3)\n",
        "\n",
        "(x_train, y_train), (x_test, y_test) = keras.datasets.cifar100.load_data()\n",
        "\n",
        "print(f\"x_train shape: {x_train.shape} - y_train shape: {y_train.shape}\")\n",
        "print(f\"x_test shape: {x_test.shape} - y_test shape: {y_test.shape}\")\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m5l3XoDybPr-"
      },
      "source": [
        "## Configure the hyperparameters"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aINE03lAbPsA"
      },
      "outputs": [],
      "source": [
        "learning_rate = 0.001\n",
        "weight_decay = 0.0001\n",
        "batch_size = 256\n",
        "num_epochs = 100\n",
        "image_size = 72  # We'll resize input images to this size\n",
        "#image_size = 36\n",
        "patch_size = 6  # Size of the patches to be extract from the input images\n",
        "num_patches = (image_size // patch_size) ** 2\n",
        "projection_dim = 64\n",
        "num_heads = 4\n",
        "transformer_units = [\n",
        "    projection_dim * 2,\n",
        "    projection_dim,\n",
        "]  # Size of the transformer layers\n",
        "transformer_layers = 8\n",
        "mlp_head_units = [2048, 1024]  # Size of the dense layers of the final classifier\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bXv1tCcjbPsA"
      },
      "source": [
        "## Use data augmentation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bRNiQkSxbPsB"
      },
      "outputs": [],
      "source": [
        "data_augmentation = keras.Sequential(\n",
        "    [\n",
        "        layers.Normalization(),\n",
        "        layers.Resizing(image_size, image_size),\n",
        "        layers.RandomFlip(\"horizontal\"),\n",
        "        layers.RandomRotation(factor=0.02),\n",
        "        layers.RandomZoom(\n",
        "            height_factor=0.2, width_factor=0.2\n",
        "        ),\n",
        "    ],\n",
        "    name=\"data_augmentation\",\n",
        ")\n",
        "# Compute the mean and the variance of the training data for normalization.\n",
        "data_augmentation.layers[0].adapt(x_train)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1WN8Aa2VbPsD"
      },
      "source": [
        "## Implement multilayer perceptron (MLP)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "INqLA2cvbPsE"
      },
      "outputs": [],
      "source": [
        "\n",
        "def mlp(x, hidden_units, dropout_rate):\n",
        "    for units in hidden_units:\n",
        "        x = layers.Dense(units, activation=tf.nn.gelu)(x)\n",
        "        x = layers.Dropout(dropout_rate)(x)\n",
        "    return x\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U33MLzK_bPsF"
      },
      "source": [
        "## Implement patch creation as a layer"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sNxPjBf7bPsF"
      },
      "outputs": [],
      "source": [
        "\n",
        "class Patches(layers.Layer):\n",
        "    def __init__(self, patch_size):\n",
        "        super().__init__()\n",
        "        self.patch_size = patch_size\n",
        "\n",
        "    def call(self, images):\n",
        "        batch_size = tf.shape(images)[0]\n",
        "        patches = tf.image.extract_patches(\n",
        "            images=images,\n",
        "            sizes=[1, self.patch_size, self.patch_size, 1],\n",
        "            strides=[1, self.patch_size, self.patch_size, 1],\n",
        "            rates=[1, 1, 1, 1],\n",
        "            padding=\"VALID\",\n",
        "        )\n",
        "        patch_dims = patches.shape[-1]\n",
        "        patches = tf.reshape(patches, [batch_size, -1, patch_dims])\n",
        "        return patches\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o_dQoKh1bPsG"
      },
      "source": [
        "Let's display patches for a sample image"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YRV7M9XubPsG",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 742
        },
        "outputId": "f0c0702a-54e9-492f-d0d3-07a184355229"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Image size: 72 X 72\n",
            "Patch size: 6 X 6\n",
            "Patches per image: 144\n",
            "Elements per patch: 108\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 400x400 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUgAAAFICAYAAAAyFGczAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAYDUlEQVR4nO3dSY8k+VnH8SciMnLPyqy9urqnu6Z7emY8tsdjBDa7EJy4ceGAuJgzSJx4IbwBJMQBiQMSJxAgYYEN8sIYM57ReNzbdPVS1VVda+4ZCxdu6NHzmxNIfD/nR09mZUT+Kg7/J5+kruvaAAD/Q/q//QYA4P8qAhIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQCOhlpY/NEfSnX16adhzUJ81e/9ybfCmurmvtRrOV1Kdenzi7Cm221KvR6n/yLVvbv/22HNP//Hn0u9Pv7Lx2HN/V+7I/V69xfekepOLwZhTZoXUq/jxz+W6sbfPg9rrpfa//+8FV/P27/xFanXrffvSXXtdi+sqaqx1OvTn3wk1SVXVVjzm2//stRre20U1iwL7Zo/eHEs1Z1cXcVFVSL1+tbv/rFUxxMkADgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQAOAhIAHAQkADjkSZraZlpdGk8ldOZnUq+3f/TTsOb5nR2pV9GLpz3MzKZri7AmvcykXjfWtKmKi/HTsGat1N5/Oo2nFzpjbRLo5dMj7TW77bhInApZ/udEqlsIg1HzxVTqdesb98Oa3bfvSr2SSvgszKwu87Amb2xJvd776telulefPgxrribaZ1YV8cTKolhJveZiXZ7FcdVO4wmlL4InSABwEJAA4CAgAcBBQAKAg4AEAAcBCQAOAhIAHAQkADjkg+LpQjwo3ol/zr/YWJN67T6NDw1PX2sHiydt7XC67QzDkpeXP5RadRctqe5yFR/gLebaQevBfvyaWV87mDvTzgxbsxEfrl8vS6nXk4t4LYCZWSOrw5q8rR3oT/biQ9vauzJbJtqHlpjw/hPtQH8Wv30zM9v/Unzw/NVD7XtyfhHnQVlpKxeOLrWBhFYdx9XexpekXiqeIAHAQUACgIOABAAHAQkADgISABwEJAA4CEgAcBCQAOAgIAHAoa9c6GhTCatxXNdo9KVeiXBy/uqj70q9XtzUJoFOG5dhzb8e/53U6831m1LdN2//XljTXsU/cW9m9vjn48/jWXYu9bqx/ktS3fbuG2HN6x98W+pl2vCFVUk82zK8pU1s9Tc2w5pVfaX1yrRnjlL4Q5e1NiIjfBRmZtZqxZM09fqJ1Ovws8dxUaXds6+PX0h1eSPOlnZDjjQJT5AA4CAgAcBBQAKAg4AEAAcBCQAOAhIAHAQkADgISABw6AfFp9r6gOvjuOVoN15rYGaWWLxmYPtyX+r1ci9eBWFm9tnJn4U1g9621CvP45/VNzOz1euwZK2p9frGG18Ja55edqReZTs+AG5mlgziuu7u+9pr2j9IdVUhHBS/p12npBsfaK5MOyjeqkdS3Uy4nGWiHRRviAfFsyr+Du9vvSn1evkkPih++OqJ1KvV1O7H6zJeZ3FypA2O/IFUxRMkALgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQAOAhIAHAQkADjkSZrk1l2pbuNmLy6qHmmvWSzDmsZSGyPIrCvVTRfXYc2t9ltSr68Wvy7VjZL4vb2Yfyj1GqzthTV3drVpidki7mVmNruYhDXNjVtSr+5oJNUVFq/QGL0br1IwM1tbj8da2umG1GtVN6W6QR6vg0i1jQWWpCuprteO/85mU/s+vflOvE7k8OyZ1KvsaTE0O4tXhdSl+KGJeIIEAAcBCQAOAhIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAIc8SWMW74MwM6vX45Ps6at4CsLMLLF4kmbZKqVenYb2p76/8fthzWC6I/Ua5AdS3VYnnj56MIonfMzMFu1PwpqbI20S6PrJQqr7/l/9fVjTF/5GM7P3fyXeqWNmVm09D2umRTx5YWb2+b+dhDWrS20qZ5hre3A2RvHETd4Sd9Jk2nNO2Y2nZEb72mtuD+NJoJ2Rds0vXmnZspPdDmvyrjbJpOIJEgAcBCQAOAhIAHAQkADgICABwEFAAoCDgAQABwEJAA595UIm/pT59eu410w7GDq9eyesOb+xL/WyaSaVbY3jdQSvnp9KvR6cfEeqy38Uf7arrva/rDWKD9Nejz6VenW3415mZkefvwhrtg+067Tqrkt1f/2n3w9rzo86Uq9uP14fMOhrAwlZHh86NzPb3ooPlLc62vsvK21NwrmwsuDewYHUazRshzUHe9+Uek0H8f1jZpY1WmFNUcdrJb4IniABwEFAAoCDgAQABwEJAA4CEgAcBCQAOAhIAHAQkADgICABwKGvXJi8lMqy1VVYMz8/knr9Y7UX1jz48bHUaz7Xpg2qND6tfz3VVhFcT7Tpi5Pj+G9oij+rf3YST3Is6rHUa/+du1Ld0dP43jg90qaPfvbhT6W655/Ef8O81lYuDHfi1R4vau3+GQ5HUt3jJw/Cmn5/IPU6Ea65mdneXvx92trdkHp98ll8nXLx8eu9d+9JdXcO+nFRra0mUfEECQAOAhIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQAOeZJmVVxKdaej3bDmLyZNqdd3/uaHYc3OLW1vyt7tN6S680n8d15daaf1Xx690upePA9rxpMLqVc1W4U1H9y7r/U61CZRWsL0UbvWdgJlK+1/9vp6PGVyXRZSr9Uq/swG/aHUqyi06am6inenZKn2mXU6XamuqOP39vEnP5F6zcbzsCbPtPc/n2vXab6IJ7s2N8TdWSKeIAHAQUACgIOABAAHAQkADgISABwEJAA4CEgAcBCQAOCQD4r/7btfker+6aN45cKD0/gn7s3MtoQDzbOF1uu734sPnZuZTRfxAdj5dCb16g60n8w3iw+37u3sS53yJP6fdzSZSL2uKu0AbyIcyC6z+GC0mdnJRFtncSr8DY2GNpDQ6uZCTUfqdffNA6luuYo/s3Zbe83tlXY4vRDWRlxNtHs7acbXcz6bSr3my/g7Z2Z2JKyWWC20A/0qniABwEFAAoCDgAQABwEJAA4CEgAcBCQAOAhIAHAQkADgICABwCFP0nz3MD6Fb2b27Dg+7X52/lrqNer3w5qfPXok9Soq7f2vDdfCmob4E/ftPF5FYGbWacd156+19QdjYaol14ZabJVqt8d8Eq+gSKfaxFOjvSXVmcV/RCmsGDAzu57Ekxxnl4+lXmm7J9WVwpRSu9mWep2L36f9O2+FNU1xTcLo4CCsORNXjkzm2gqTs/P4ezIV7zMVT5AA4CAgAcBBQAKAg4AEAAcBCQAOAhIAHAQkADgISABwyAfFH36qHcj+5OMP46I8PgBuZvbj8adhze1bb0i9ykr8WfoirkvFQ+fzuXZodTyO11TMipXUa3NjPaw5f6Ud4E3HY6luIfy0/mKm9aqb2mfb7cXrLMYzbbVEsxPfj708HiAwM+v2tHs7SeMD2TNhLYOZ2freDamuquN7aPv2gdSrGF+GNW/cuSv1ml3H97+Z2cX5cVizub0r9VLxBAkADgISABwEJAA4CEgAcBCQAOAgIAHAQUACgIOABAAHAQkADn2S5on2k/NJqxO/aNaUeu3cvBnWFGku9SqLeNrDzGxeCPsIhLUGZmZFov3/qdL48xhsapMcizKelhjs7ku9WuJnq0z59Pva+6+b8YSMmVkqrLOYVNrEUNaNV2hsbG1Lvc7F6aPbb78T1lw9fKj1eu/rUt3hg4/CmkS8Z+s6ruv14ywwM1stte+mMplzfaGtJlHxBAkADgISABwEJAA4CEgAcBCQAOAgIAHAQUACgIOABAAHAQkADnmSxhJtp8twI96PsbauTVUs54uwpjvalHqdvYx3aJiZbe7HO24OHz6Qem3c0CZWGq14J0fe1C7VeDYTquJ9KGZmSUt7zc6aMP2y1KaP6lycjErj/+1r29qulmUVv7d2P971Y2aWZdozR7GIX/Pg3fekXs+efCbV3Xnny2HN0WNteuftDz4Ia44faXusBus7Ut3k5Cysya0t9VLxBAkADgISABwEJAA4CEgAcBCQAOAgIAHAQUACgIOABACHfFC8N9IOcyZpEtb0ByOp1ySPf4q90dAOPQ/3bkl103IZ1ty6d1/qtRTWH5iZZWX8k//NpramoteIf+a+rLVD23kqrJ8ws62t3bDm4uxU6lW3tIPiWRavXMi62kBC1+K/szfsSb0217QD5dezeAiiN9qQeuVHL6S6LIu/mzt7e1KvyVm8WqLT6Uu9eg1tiGP/a/H36c272jVX8QQJAA4CEgAcBCQAOAhIAHAQkADgICABwEFAAoCDgAQABwEJAA55kmawNpTqUmGyZbKKJ2TMzLrDeJ3C9ekrqdfopjZJM3v+LKwZvhWvZTAzuzrRpkcaw3gqJM21SZqsjqdCFrNrrddKmwTa2IyvU28grGUws7PxRKpb1PFUSNaJP1czs939+N6op9r7aq1rK0Auy/i+nV1p1+n+l78m1T1/fhTW3FyPp6LMzNr9w7Dma7+o3T/797U8GHTje+joZ/Ek3BfBEyQAOAhIAHAQkADgICABwEFAAoCDgAQABwEJAA4CEgAcBCQAOORJGsu1LO1txJME02m8z8LMLBXWzbQHbalXlsSTF2Zm67s3wprFStvp0mhp721nPd49clVoUwmJsF8lSUqt10qbSiiEz7ZItfun3df2mDRMmKQZatM7o914emS1iHfImJlVtfbZ3tiLp7Gmi7nUq661e+Puvbjm1ls/knrdejv+DoyG8X4kM7NOpe3eefS9eDLq80+upF6/83NSGU+QAOAhIAHAQUACgIOABAAHAQkADgISABwEJAA4CEgAcOgHxRfxAWQzs+n1RVjTbXSlXldX8UHZ/mhL6jUea4fTBxtxv+lE+/n9Trcn1SXN+DJ029rKhX4nPpx7sdA+i6SnrSxoVvGhbfFsvfU62uH6UjiQvbm3L/UarMUHyvO+1uvV559LdYuLk7Cmn2nDDVtt7XD07s14hUO3ox2urz5bD2seiWsqLq61A/FXV6/Dms3eSOql4gkSABwEJAA4CEgAcBCQAOAgIAHAQUACgIOABAAHAQkADgISABzyJM0Hv5VLdWUjnnBIC22SYzGNpyquXs2kXpNE+/n3ViOeWCna2lRRr6dNDJVZ/H+qJ04MLSbxVMX2jZtSr/FSWzNQn12ENWs9bUKmGmg/v5934355R1vfkDbiz7/T1e6f0e62VDfvx++/+/Kh1Gu7r01ZPfpJXNMU1pyYmWX5aVhzWj2XelXC529mpmztyOfaZ6HiCRIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQAO+aD48A3toHhSC5mbxT+dbmbWz+KDys8+jn9G3sxsOtX+1Otz4QBvdk/q1cy11+zu7IU1y0zr1Wsr16mSetVT7RC+CefmJydHUqtuT/vJfxvE6yyagzXtNVvxNZ+eX0q92uJqjEY7vubdair1Gi+eSHXzVbzaYFFp17zVXYY1SRXXmJmlWTxcYma2mMZ1F2Ptmqt4ggQABwEJAA4CEgAcBCQAOAhIAHAQkADgICABwEFAAoCDgAQAhzxJUyy032IfJqOw5ujVidSr7sWTNFVH/Cn/qpDq8l5cl23cknql4k/+WzOevqjn2voDy+PrVFmi9Uq0/5/dUbwmoRxrazaydkuqq3NhYqXQVmNMq3jCpF5p90+jra3ZsGn8mpOJNomS1Np7K0rh3m5pU1aLSdxrVWq9mj3tOmV1fM0XU/HeFvEECQAOAhIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQAOeZJmLe9oDS/jusFsV+qV9OKJm85Am6RpNrUJjetVvPdiNtYmHJqZNkkwL+IpkzzX/pctpvHETZZr+4XE1TVWCJM5q0qbcKgSbWKrUQnTF+L01GK5CmtawoSSmdn163OprjGPJ2lupGKvVLtQ1UqoE695Xsbfp3ymNZsutN07RRlf824p7lES8QQJAA4CEgAcBCQAOAhIAHAQkADgICABwEFAAoCDgAQAh3xQfDERf/L/Mj7MORRWDJiZjRvxofNmc13qlQy015zM48O5xeETqVfVf0+qWyzjA7VVrR1ULqr4oHtt2k/cp+Jqhul0EtZMptrKhe5cO+ibN+LPY7rSDvQ3hFUElWn3T1Joh9ObeXwNBpV2ULxuatez0YzfW0ubp7DtMo6OE+Uwv5nNy/igvplZnsX3Y5WIOSXiCRIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQAOAhIAHPIkTW+hnXbv9OKf8391caG96HY8JdPs9qRWWaatjBhsxqf/p8eXUq95pk2/ZMokR6FNtdSr+DotU23ao5Gpqxni6Z35JJ62MTNrCesPzMzms3hlQZpp//8T4aOdz8WpnFp7zYuTs7BmnMYrR8zMtm5q02Tvv7MV1iyX2iTTnc4grHk01u7ZeaF9N9M0vh8r8ZqreIIEAAcBCQAOAhIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAIc8SXP65EiqGy/izE1b2sn53X48IZA2tGmPvNGX6kab8fTL9Y629+Ll4aFU1x0N4yJl3MO0CzpfapM0SUe7PSphD0tD3G+TJuItmcb32WqpTb80W/G+mWKuXfOyjqeKzMxu78STNJ0ynhYyM0sybUqpn++GNQ1xEuXe3bfCmmf/rn1mtbjHx5I4N3JtsE7GEyQAOAhIAHAQkADgICABwEFAAoCDgAQABwEJAA4CEgAc8kHxR09fSHWdzlpYM9jVDoo3hQPlDfFgcZprB5VbzfhA+c07+1Kvanwq1V3P4v9TZdKWellTOKgfb5X470Lt/2curJbo5tr6CUsrqSxL435lrf2hZRm/Zioe1O9W2uHu4cFVWDO6EIcgSu29XZzFr9lrat+njx8/DWsWhXb/ZD3tcH13FP+dk6n2+at4ggQABwEJAA4CEgAcBCQAOAhIAHAQkADgICABwEFAAoCDgAQAhzxJk1Zali5X8c+/nx2fSL0OVvHbS5otqVfe0KYqmo14QqO5uyX1WhQzqW78g/gzK4VpFTMzq4TPIxH/L4qTKIkwCJGW19pLimsS6nb83jJxymoxi1cDFGcXUq/utrY+YFHHdZ3+htRrchivbzAzqwphGqupTbkdn8Tf4UYeT9WZmU3m2mfWyePr1ByupF4qniABwEFAAoCDgAQABwEJAA4CEgAcBCQAOAhIAHAQkADgkA+Kbw57Ut2z1+Owpiq0n0WvhAOkab8r9UpT7WfdU4t/1r0WD51vbG9KdS97z8Ka10+0lRf93XgdRNLQLnu1nEp106v40HC7px0Av7zUXtOS+OB8LZ6tH5TxKoIiP5d6ra1pwwF5ZxDWzMbxAIGZ2XSu3dv99VFYU5u25iGt4tUkaSKuzzDxcPrT+B6qm9p1UvEECQAOAhIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAAcBCQAOeZKm29YmVja34p9FH8+1n4g/PPwwrLm78atSL6vjCRkzs9ri8YtS+Ll8M7NmU/t4D94fhjVJeij1SsvTsOb5kfZZrHXjaQkzs84qnkQpm+L7n8efhZlZdXUrrFlb0yZRuhsfhzXtfe1aDrfWpbp2K56ySgptqmiwcUOqS1rxvT2NB+HMzGy+igtnE21iLutp99n5WZwb48ml1EvFEyQAOAhIAHAQkADgICABwEFAAoCDgAQABwEJAA4CEgAcBCQAOORJmvGFlqXPTuOdEOv3tdPudXYU1mS19ieU1Uqqyyzeo5GIUzkNYb+NmdnWjXiPzPr2SOp1fvwgrElybUIjXWgTQ+kgnlg5OruQeu0OfyrVTS/iKY2tobiUZl24t+um1KpqtKW6XHhrjZa2B6rR2pDqLI8nVq5W2u6jy+t4emosrhfKFq+lutn0OKxZb9/WXlTEEyQAOAhIAHAQkADgICABwEFAAoCDgAQABwEJAA4CEgAc8kHx6SI+GGpmtljGB61XeVxjZmZZGZbUlXbINKm1n8Kva+G9iQfFK+HQuZnZZB7/fH21jA/JmpnVaXxJB/e0/4tZHX/+ZmaT4/gQfmupHdpu9rTXzO/FKxwa/W2pV9aI1zxUdS316ja11SSlLcOaPNOGG2aLeDjDzKycxSe3i9W11KtYxvdQq5VLvZaFli07g72wpt3S1jeoeIIEAAcBCQAOAhIAHAQkADgISABwEJAA4CAgAcBBQAKAg4AEAEdS1+KIAAD8P8MTJAA4CEgAcBCQAOAgIAHAQUACgIOABAAHAQkADgISABwEJAA4/gv2RagLN0HkogAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 400x400 with 144 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAUkAAAFICAYAAADd1gwNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKmElEQVR4nO29e9B1WV3nt9bat3Oe53lv3YAyikMqVqrMTJykMiM2yYxi6VjqDF2QoIgw2lyb5ioCNpdWLiNDQ49cRaAbaFBQSJARwziDQygnCTMSrVBmmNIaa4I23TTd/V6fyzln31b+ML7r+/vu51n7vKSrkiq+n7/O793n7LUva693/77P7+JjjNEJIYQ4lvD/9QEIIcT/n9EiKYQQGbRICiFEBi2SQgiRQYukEEJk0CIphBAZtEgKIUQGLZJCCJFBi6QQQmQot/1i/8IXGDv2m2SsLthtm8vGHmApHrzd795HP2fs/3j7zca+/2//jaufj77tW+w4e7vWrps0TjeYbT/ybT9s7M985Z8b2x/A+RyszbYy2P9Lqqq4+nkdLtlx/tazHfPxP36dsYeqv/r5ut3Hmm17C3uO9z74769+/uoD/95s+7knfNjYr3r/PzT2+T9J96EIhdl29ttPG/vco8+kY/rWc2bbs7/vHcb+xJd/0djrTX3182pdm23R24QuX4xXP7erK2bbi77vTca+/Xeea8d56FL67T32HnVfa43djmmitXYquPfc9bvGfsFz/4GxiyJdq+X1p8y2U4+53tqPvi59fuRZs+2ZN7zEMR//0w8ZuywrGLeib/fGirG7+vlg3z5jz/huO9Y7P3+bsQ8v7yejtfekHOzc+I5zf+3q58ecfbTZ9vjv/Slj//GXPmXsJTyDwduHfaDkvk2bzufS4aHZ9nf/m6cZ+3c++wFjr9p0v1ebjdk2ySE0tj2mn3nK9B4xepMUQogMWiSFECKDFkkhhMiwtSYZ3Tj5l23xIBL4md8VvdVhqk3SHirQMJxzbhjtMY2gf8VA4icRSGd0+H3a5klbcTHZIc7/P1M41ppwX/Z6jCPpUGPaHmb+T/ORj/OEz845P55sjz3fa0vf2WMchjSNJkWlSJN0EfY95MeJLd3fTbLjYPfLw+J1m6tz5Qt73YomnU+5sPeurK3mWlRpuyfd94TR6ED9ydvofnuwp/qlpaoaY9dNeo5itCJtyFyfcebi8TPYD2nfwdvjHyJ9d0zfHcb8OJN5hTY9nzzl8Lr6yTWeR2+SQgiRQYukEEJk2Nrd5nAE59KrMnt5zlu3w0PoQjl2LsfO4ZGxz5y/lH57yob8rHcXxu7AHou861NX1l0Z6+HYz8455yluCe1itO7XcSzinrGHCCEL5HJuun1nv5yuezXmbxeHcoQeXHXyQcqOXLlNOqfuKH+PjvbtPRrAhRlpSnlysVxM5zOu8+OMl2xYz3gx/XY4YjfPnl8PbiBvY0Jtr8Xi3M7VzzvX21CpnevOGHt5OoUIlbWdj8cS6R5GuGfjjMwDz1UzM+1On7nO2A243y3dv/HQXmcP0lPb83NvWbf2t3i72bVld3sD+54bh3+Ld5RDjVj2KPAay90WQoiHFy2SQgiRQYukEEJk2F6TjBSaApok/wmeNckAOqQfKUeMWKxsutnpi0mjC+cO7DCPtPqQB91tLPLaQ1PaU++qZPcN6Wpr0rRASiv8fNhHMy6NPUDYUBzs9Wij1SRjnwab1SRJ0wog4/D/hkVvr0+Ac+pWM5rk4crYHlLrfGV1uUCxRqhPx82MJnmFQo0uoZ7JoVP0W9AhxyGvd4Xa3sPmbNIkOS1xec7azamkNxdh/nHyFDIWcS7EvCaJITV1nZ93e6fOGnsBoUsrmg3tSM9VkbZ3s5okheVlHm8OF+rgvmy6/FzoaMejCSukEC4KPao8zM9v4L1Qb5JCCJFBi6QQQmTQIimEEBm21iQ96zqYRuRtCpQjvc9VSQ+JLh/gVQS7rwXEw40HVrfoKZ4vrlP8YR9s/BZTDlaHcQWcz9L+39GPNq6s61JZpzjaMk3H0Y9WwxtBZxwo/rTz9py6Lh3n2OfPCcuQOedcBefBOk1oSGctIU50Jo2P0xKDh1S0YLWjgtJZS4hwK9uZgTaUeoinTwc5yTI1x5fXpz3FSZr739BvSQqMcD6jy+vtzk3vtzmOic3vMPjM5VM6A+UaFlXa+2KPUi2D1ZE9XOdVm5/fh4Od2yWuE3R7c5rkqrV/i2AONrasnplWdCkWxY6xqzKtKeUWujGjN0khhMigRVIIITJs/+5J7rYJZSB3O5K7HUtwt0tyzYmC9rVYQeUXcrc5fW6EikGhyLs+U3cbZABKWew663Ksj1Il9tHlXWDnnOvI3XZ9OrZ2sK78xpFr3yZ7HPKuTwjk2i4hfTJwpRv6MbjqeUfOuZ7CQoqQ7Ejudk0+VwXpZX03MxK54yPa9FNOu0T3m6vRMIFcao/u9oJCdii0zLjbfgt32+dDasxx8TsMhJv5GXfbT9xtrCBkJa+apJfuStr3rLvdWzc5ZNxtruTTDen5PdzYyuTM/tq62wVWRKLQqSbY57eCyVDPpCsfh94khRAigxZJIYTIoEVSCCEy+Dgp+SuEEOKv0JukEEJk0CIphBAZtEgKIUSGreMk+5ufZv/BlPuiOMlo45RiBbGEpY0rbN7968ZuX/FsO84ypRgdfqstpX/xsWeNfeVbU0myDbVgeNzffLGxf/8/vM2OiyXrKcft8v7X7biX7r36OY72fF74/R9zzF3/603Gbup0verKllErSxvDtoGyZJsDG0N50w/eZew7fuNGYz/49Qeufh4psHBnz6ZuNbvJDo1tk3Hrkz9h7Ld+5pnGrhfXH/vZOeeKtY0RLY5SKbijr9vYtxc9257PG257irEPH4KYUer4x+X9xzLdz6G0sY3vese/MPatdzzd2Kcf+6irn3evP2e2LXepVNoipfRxs8Rn/I2bHfOxP32fsbHVQKCUuZJ2WMK85DjIJz7GnsOn7/mIHQdiOD1dO64zt9lPc3pzxcYiP+3xP2vsX/vcHcYeMK6ZWpPwnz96SM89Wts4yZc+/XZjv/kD9vnFGNKCEjpPL2wJxbPL1MqiphjRG2+c3iNGb5JCCJFBi6QQQmTQIimEEBm2z90mXSdCq4CResqO3BoA6kt5yqtkvOdDSr+N1L6yv3LB2BsoB3W4m8+R3T/4qrHXkH+8DlaHefDgXmPff/TnyYjzubj3b75s7DMhaSTXN3/dbDvVnDV2Ae1ry5mI1sUZ+39eCTnj68Hm2K5KqxX2VcrRXdb5exQWViurl0mXW+5YzW7o7D3rNuk4WtIrmZFbB4CmFTkxmHO3oQVHscxP83rXaup4PkVNv6Wc6RHqtxUzJdmccy5QKTzcXyTdeKS6bAO804SZucBl2/DyFPQMesrrd7i5yeeId8Heo1WfdOOh5ftnzQHu72pFtRSIoyPb1gSvNZfC4/tQYO52Od8CmtGbpBBCZNAiKYQQGbZ2t7mTG7rbQ2e3sV1CJ8KimFmXOY4CXJDQWxcirMjFOoSwpCLvj3Sd3X4IXQr340Wz7crmAWMfDZfTmFuUx1q7y8bewRApb93RktxGlDlClS/ztLtj3ajTI1RkpvMdInWELLDjYb6cnats+JCHkKFisWe2xZpKbRUQ8uTz7nak/8Njxt0mxceV0E2wPJU/n2rXumBFA9eCZabAXUOxa9/8O0cglzpC+MrouAzbyTZ/l+FkY9t50d77QB0/sVJgXOTH4TmJpeA2VCKwp3KLWOG+6/OVybuBpDY8LD7E1v5DDxJHWagyuRBCPKxokRRCiAxaJIUQIsP2DnpHLRmGtL72a+ou2NLa20Dowkz19EgpgZhOFbhMe2+1pEWX7KN+5tTGs8Zct0mHvNRZTZLDZwKEUGyjcZRUMh7bLPhI7R9GO1YB3fXCjM56qiHtLZ69+vGws6EnR3Q/Owe6YmnTuphY2jAfVyXbV3ZbaKwOVe6k8ysWM+FTdN0wrGVS4Y+bGkIIUH3Gpn4yrEl6mK+Or/mk/QKk+810AnVu0mzRDbh7OgfWJEcMAZrRJF3k9x9IaaRtrEnWZQqBqhZ5Pbeu7bX15aWrnztqbYLhQc4515rwsPzcHuk+jBHbjdi/C3S9neuHY/p7wxZRWhP0JimEEBm0SAohRAYtkkIIkWF7TTJwOTTs2Uml0Tj1ENqVTjQeYtr9EzTJkeK5evvlCjTJYly4LKON52v7dFwHa5sC1VNsXAXXYrGFJrnwVqdbuDR25ey1KuLJ2kt0+ZjMBbXy9cs0bllRi9HO6mfrmGIfY53XJIvSXjsfki4VaZ640pZdKxZn0+cm30Y0UAqZhxhbT+W9SFZzJcSMNnOa5J4dp4QWswVN5WnrWkyPyw7zl8dJohi2u410Et6zPg/nP6NJTn7r8DjttoLGDSX+DSH/HrVYUnvaRZqDnkIfu43VKDegv4dJOrJlpM0Yc9mP1Fqa4jEjpD9O0lm3QG+SQgiRQYukEEJk2N7dPmVdsABpidWSXMbB7rZYpPQkX+RT0RylbXlwNycuFlU+hsiaSZgD4ykYAwtb94N9fQ/0f0ldJFf+lM+7ps45d118rLHPxFTtei/a39fOuqdtTNW7Nz2l+BGRXMGySse5qMhvXNrUwgokgNHbCvDMsiYXGqbRsOHKL+T2QSX2YmH3w5QLK5lUTXLtuDI5z+QaUhGb6/LSS3PaXpt6mVzTquIKM+Sa++0rXDnnXKTq4/ibin4fJuFw6G7nKTgkCuySHo2KykuhpFBwmiyx3LXHfOpsmlebwcoch62VV2IL83lGqxhru33o0gPbt/a5GKlKvTOSldxtIYR4WNEiKYQQGbRICiFEBh8n+V1CCCH+Cr1JCiFEBi2SQgiRQYukEEJk2DpOsn/dPza2befA8WEU3xWhfQF+ds6Vb3yHHee1z6d9pZjFdmlT3g5O2XS/y2eT/dAjbOrcDT/6EmP/9hd+xdh/fvRH6fPhH5ptTWnH3avPXv18tvgWs+2WH/h1x3zi86809tn6+qufl7v2FjRLGw92sE6tI/bXXzfbbnzch4z96T/6GWNjetlY2ZjErjxr7B5SJ7vexlA+/bteaOy7/90HjT0M9bGfnXMu9DZmrRhSfOP6AdsW4/k3/qyxb3/7i4y9ejCd/+goHrOy4+x+Z7rGO//pdWbbi254q7Hf98e3GbtusBQetXagcmgB5j7H3v7Ed97imI//2fvtYcP+y0BpmJTCaDsW2D8l/Nh3PM3Yn/kLOw8xTrIuKS6U0lmxxUpBQZXfc/pGY3/+a79m7P2DVGbwgYceNNvuve9rxr54OcUAe0rvfdtPf8zYL/vI043drlN3xXZjOy3Ggf7MEk/47Jy782Wfc3PoTVIIITJokRRCiAxaJIUQIsM19FckDQhzLbkmPfdogNaR5vNxRLvdY8l+KhU2BmrPCWXYPCevEhVpIHvVI69+vn7xn9nvRqtJ7sSk3+0NVu86jsX4aGPXY8rXXlAub0MtTI/qdM4Dt3ogNjXlKJdQioqr19VWdwwu5Wtzu1mGKlG5/Ycuw2dqdTFSPjkoa0WXz+Ovduz51N+edMZY2nFiTa15z2Kec/66dYc2p7i9DO8Og71wfrDHFCKU56O8bved07EO/sLmGdeQG16XVLKM5jDmbnPJNvcd1mytHOgKeF4j5WqzDSn/Ljb0HkVp/aG3x7yA8nY7tb0eOw2V54NW08NMZ2bf2i/UI+jG3v79wZWs5Sabdd5t0JukEEJk0CIphBAZrsHdtuXDTAlxKiflSipVNIJ/1ubdbc8lsIyLTe51oCre4G6HmeW/Ijd3r0zu9iMaComgrow1hMjssv9xDIuB3O2YwnEW3rpzNLQLMV333lGpZ6JtrCs4QFm6mkpN1Q252zGdx6bN+z5USc7tP5hCOb7+Z/fTfu24FZT6Pn0qP/3Y3d49ne5DXFLYR2M78Q0QxtK7fIk5drc3V9I86lf2/oytdd19n46xCfkK6M45t/8X9veLJrmrA938QN0i0QwzpcU2NrrK/JarfI9UtT7ugXu6m3+QQsdyUbpeS3Kvl429losqHdSmz68LobNzMoAkFJyVw/i6oS13WwghHma0SAohRAYtkkIIkWF7TdKzCSXRqVuZ6yk8AdoO+J6+S0TSDMY66TwdtRxod6zdLdJ3h5lS+p46LdZD2tcOySPjxh5T3KTfHvX58BLnnLtwwaZi7oekgV06sLegsqfkLsXUufEyl6X/Lmvu30+3E1oldLXdNjRWpysLSPPqOKbLwmlf7VHSSg8v2U6TgTrxNQvoNLmT1/BGjiSDnLLLD1oNcv/wvD0muL9tT5P3Bmv+X1+6YuxhAwOTHs0tRAJo5lXIz23nnLv/qxeNvVika7dcWM25mKQLYquIvLZ2/332nHwmN89TONHuqXRfdvYorMxGx7mvfcXeh1Cl67Hp7DNYrc8a+zSECrZFXjc+U1H7GIeaZL7LJP6B4huQJPUmKYQQObRICiFEBi2SQgiR4RvWJFHX8BQ4FzlvrQPdbiYeaqSUxgF0yG7Xph9t2F6k7w5+pr0n6W5lB+1XKfZrs7a64+awhW1WkzmOBx6wOWID6Foltywluw3perVF/trt30PxYRALWVKM2qaxKYElxFH6Ot/qNVIr3w6uz9G+jTksuJUtpNq1Y/4ekWzs1tBG9P57rM5771dsfOZmBTF4K5rmtgKb+7M/tPsKEO9YFpSyR3YB9ySEuXbJzt371QvG3lmmsZYLq9HytSvK7eP97rvXjjPCnGsp1nOgZ/L06VPHfnbOOfcj1vzqf7DxqhiPW9X2+MvqnLFPe3i2y/xzdLa2vw0+3VPvZzR0uFRxthnvFL1JCiFEBi2SQgiRYWt32w/2FT1iiiC5255DgtrkgsU2/1q9ot8ewDJ+mf60f7Gzrs/+EbimM57P5Qv2OHKFilYrmxK1hnFWq/kQoItX7MG0XTrHSGmYkcN8oKJJnKlsdP4vKG0RXEFf2hALX9nzLyGFrN6jVMvHW/PCfdaVuwxVgA4vW/crVHaKdeDa+ZCXD75+nw2X8VAF6WtfsSEuX/uPNvSoXYO7vc6/Czz4VbuvCiovlZWdjyVV6kG7KPJun3POXbxgz+mwTvehrmy4UTlxt7dXx+6918oPA5TZGbjkDkXsdSBrjEP+2p1/kO43TFE6HbegaKK6wjTimXPrbGxcAfOqousSKaRpNM/UtTeH1ZukEEJk0CIphBAZtEgKIUQGH2O8diddCCG+SdCbpBBCZNAiKYQQGbRICiFEhq0Dr4bXPN/+A8YecVzkaGPyRkjdG1c2ba2587PGvueVP2Hs+x/xqKufzy9titSlwgZeHUB5tG60MYXPv+VWY7/rV263xwi5S0O0/3e01HJis0n2hlK83vS6Nzjm5a9+lf09/Kbd2NjGjtsDQGwkl8e6693vM/bP3HyT3VeX9j1SF8ox2Fi5AloH7JyxZanuetOdxr7ljS8x9oP33H/sZ+emZasKKNnWLGwg3Wd+8zPG/u+e+SR7zJDuev6+h8y2i1+38Yc9lHPjuNc/+9M/NfZ3ffffMnYNqbAcm8jpgNgaoKK4xi/8Kzu3nXPu+37kx07eH+27KjmtL9kjlWz7l7/1Pxj7+3/sicYeIJW0pnTBhtp+XP+I6+HzI8y2t7/xdcZ+0a2vNfZmA88+xfxSyKxroHXF3q49hje82j5Hb3nnm4y9uwvzdZfSWz11VYW5H6kFzLOf+mo3h94khRAigxZJIYTIoEVSCCEybK1Jxt7mH0fMgWS9a7T2EWhJR5Q3+m00zn1UluzPD5KeeZ7aKBx6O87aJW0ixnye84XzVhsd4f+Lkf7v6Hp7zC2IXJiHfRKXD6zuuF4n3WZ1ZI9jtbLXGY/Ez+Sd3v9VamGwOlmT9MFqMyVokrtn8znV5++x4+yfT3nT7RHn+FNpMSxhdpBvkfsQtToYoK3owUX7282+PZ8etKee8+EJo6M55yJov6GnfGyaVgFK+w0DJSsfw5pK62HZvJHaYnDudl2n/fczJQcvXbLXDjXMJbU9YZ1utUol2w5pfjKX923O/OEh/P2BjpE7TizgfDZn83P7ymX7DHpYusrSPq+BytlZrVel0oQQ4mFFi6QQQmTYPgSIwnoihHa0BVXyphpJ90K3unupIji72186tK/K9+wnt+GosK5KbGwF7dAkN6GisAaGXegOJIKOKm+3FEPStsnFXm/mS6VdunzJ2Kuj5FIfHpK7Te4NShcDV3wn7n/ga8buN+k4ayrxtbe012cBpar8Kj+OO7BzIbTpegX6f5edmwK2h5j/P7qiMC7cd0OhOQOFEwVwsf2Mu80lzjBTl5N2y5DpYLiFKzeOdodxxLGoiyH9FvdfhHxZNj4nDDXiouYDhRMdrtJzFq3XPuHSZfuFzSrNjUjnyu42hrvNJUc/8KAdp+vTsz/QOHXNsgXMT3VLFEKIhxctkkIIkUGLpBBCZNhak2wp1Qeruh+RPnSlaIz9FfgT/Z+QlmKTtJz7MnV3+NqlFGLQB7txuWu1s529dIzLvbzIsW5t6M4GdMcNhfVs6LvYbW69zoexOOfcZdIkj47SeRweHNI2a3d90ni6Lq9/PnTepuphV8NTje3Et0M6skcdZ5UPaxo5dKdN172g/3dZA7KaZF4gKng72CVpbtyZz6bN5udCmGiS23+3gOsYwvw7B6c1YtpmCKTh0f6KItlz3RI5pXGA6zE5XwrTwjk9F+J25YptfdGjfk9ScKDJgM9R1+d144fO246W9h7Z89lZWnuxAC13vsPGBL1JCiFEBi2SQgiRQYukEEJk2FqT/Mp1tmTSBiTK8xu71j60strDPZBp99VVXnu44qyWEvZSebSKym45SttaQxrU6rJNl2Luu//rxm7hty2lU3WkSXag0/QUb3kcnGqIKWUjxfCx1oRtRms6X2Zvz5aS86DF1aRB9hSFdwTn2M2kPx6sbZwkXq9IMXeRzicWmC6Y19XWdB9auNYr0srWZI9wfvkZN92OJelCwbqgFbXqOunvi4XV4o/j3LlzdmwQ1ziGknVF1F3n4grP0DgYS8jnxCX4TAPWueYuNK8cPA7R0d8xxsmVvvqp5Da3RE8xwps2zcEjer5iJD12gBTGmbbMx6E3SSGEyKBFUgghMmzvbp+73tiHm/Tq/FVyde5b2VCVC/A2fHGVf63e91Q1eTdVLGFXlN0TdLmOjg6y49z3Netu4+t8R6/+PaUlors8F4rh3NTdRvgcOBmtKpO7XXFpZ2Jv17rbxouicXpORYPwonImje+Aw55MuA252+TKYXWmfsaVY3d7Dfdhzt32EGo2F5ozkttfYAofhawFCnermnR/9k7Z638c1113nbFNFSC6R+zaF8a1zc+7M2dPdrc55Gcku4VqSyw9TaA0TQ8VeEby1WOkZx+es368Nnd7DeFDJT1fXLUdd11sEabF6E1SCCEyaJEUQogMWiSFECKDj1yfSQghxFX0JimEEBm0SAohRAYtkkIIkWHrOMn3vv0Xjb3apLil+8/bOKUHLtiSZlcOU4rg5UMbv/ivPv3bxv4HT3mKsfegsxunvK0oXm8NXe/WlDr3rz/7L439vd//BGObUDkub0/tHNDmGLw/+te/75jH/cAPGhtjK3uK7+t7ivczB2nl4//j33zB2P/VDY8/8beTdEGKycPvcjzmF7/wvxn7hr/3fXZfQwefuUPeySX8uSvlv/nCHxj779xwg7E3UE6LY0/XGzsXvIe2CnSP/vxPvmzs/+Q//y/sbyFtr6C2FyWlCi6WqQTd2bNnzbbPf/qfOebGp/8jY2NMH7ch4DjJEuIkORbwt3/9bmP/6FOfbseBXRfUXqWg2M8e4xcpTvKzv/FhY//QT9rzQQb6bc9dKWEcLn33+U9/0th//8k/buydneWxn51zrqHzayCdl5/X9/zyW489dkRvkkIIkUGLpBBCZNAiKYQQGbbWJO/dp/xOaCN64dDmah8cWU3yEPSj9ebkPGbnbBky55wbMvrIAbVjPYRxp2WZLC1pfyXoFiVpNNy9E6vsBy7fdgxVbTUS1CRL0rzGwY7dQcvats2X0u8p59zkzpKeGdiG8xhm8oJHylcfYF8DlY7zVC4L27v6kJ9+nEIe3PbtW01+8kwoMOex4/fbSSk8q6thqw9f5EvZOefclcOT84wjnTC3WcB2tsNMabFLV6jdAVyvHcrxX3C5M6CYqU2ApeKcc66Acm48FzZcrm+D7WfzOeLtYOd+0UHbjLW9Tvw3hA5y/lmf3ga9SQohRAYtkkIIkWFrd/v8JetCY6ezS5fsqz13B1xhaE5r3RXm6Mi6Ix7WcXZHenqtxlfpMOMmFBTKgWEQBbkFkVw7D+7YnNv3/xyYteH37Aly5TR0ZSduITGpuoauBZePGrmMFbjBM2Wr2I3C6uwDhQBxSJCLWGYuPw7OMeecGyGsZ3KqmY6HXBqMuZa8XP4uXkYuDXYcA30F51aczBM7tzoYjMOF5sbxJvSKSqXRceOzEEgqYurFgn6b7sNYkHxAz0oPz+Aw0wm0qrjqe9pXRzIcu9RoTjpwboHeJIUQIoMWSSGEyKBFUgghMmytSV66aHXHFtLALpx/yH730kVjoz4yzGgC+wdW+1xD2At3C+RQnZ2dvauf5/ShnZ1dY1sJk1oO0G9Rk+T0vuPg79i0r3xaIoYvdDNaIbdKqLG7HodFjFYbxpAS3+fvEadSDmAPdPwj26BRRp8P+1hxuFiA86HvlqXVzgYIKeHQKIZbcHjQpFnf4u82TdLKOBzmOLgDosto56xA4zSaa0lRLW2qHmr0gcLOHKWOlqBD1gu7H4Y1SdS+C9JYKwqRQl18TpNc7p62v4V51Q/2txyIVWTC+7ZBb5JCCJFBi6QQQmTQIimEEBm2dtAvsyYJ8Y77+/tm2yGVQ0MtyXw+hpbat2KcoCMdpm6sXoLpfxxTydQU/zXmYtBoX6YEWXaU6b55/31PWiHFIKKeG2d0KF+c3P40eroe3CoUddYt2uSan8LnyblO4kCxtWkevg8etDNu7cplxcYO4g/ndGOOqwPNjmNmuawY6pCB9cYtxjIxiZy2x2mm15Be1yys5h4KaJMbOU6S5jccRzWjSZZUlmyEdYF104J0Y4yB7tr8UrSg42hbuL8tHz89B3Acfos0YkZvkkIIkUGLpBBCZNg+BIiqimD6WcuuG7kNHsroYMXo46gojKJZphCDurHbPLk3mNY1FwKUS/9jN2/SUBLsbVLRJqEc4LCP9Pof6foEqBIeQt4NDuz64P+B9NuyodCNOp1HwWWPiHKxY2x0CiO5iIH2Zapiz4xTUZgWSjXsmk5S0VwKUWP3kuF5hHZBskxDNs7XbWSKDclJC/h9s2Ov60iV9yOEWhUzFZQWO7bSD4a+bNZWDuspVdikqM6m3fJ2cOtpbnPFK9w+jvm5wGFLiyK5383Crgv85odVgcYxH2p07NjX/AshhPgmQoukEEJk0CIphBAZfJwIbkIIIf4KvUkKIUQGLZJCCJFBi6QQQmTYOk7yux/3eGOjlDnEgbZR2XYP8W0U3/V//sH/Yuy/84QfNvZyL8XKcQoUx+Bhohu3Dfjcpz5h7Cc88UnGxkpikzhJKlGGNiu6//b3/rljvueHfsyOBT+adIektMRQQYpcZa/dl37vd439X/6gvXYRWzZQOBt3eUSbS1x98bOfMfb3/n17PqujFHe3PrIxePy/cAHpcY7mwr/7g39r7L/5+L9HP07zyNO95xjFzRo6dK5tybV7vvS/G/uv/9ffS8Ok42qoFNhyaWMZsXwfx95+8Xc+6ZjHPfEpxj517mz6fPac2bbmrqOXU6wytzP43EfvMvaTXnirsSt4dg6uXKBxrhh77zQc0xl7TB99y+uN/VMvf62xR5jPJaUHTuIZIXa3XdtYzbvv+CVjP+3FrzB2WcB8pbnAZdc6aB8z0rP88Tvf4+bQm6QQQmTQIimEEBm0SAohRIZrqGVOZatATygDaw1WIyhN+fR8OamadAtTLp91tUwp9n6uLSoJSJj3zCX2243VS1rQuLbpUMn5vwWWbqIcZG5/YMp2cdl9HofaW4zDydrppM0tXueZMlyOy5Th/a3p/tLA5nLN5G5zPjaWimMNktux4v2sZ8pjTfLYzf3h1sP2XpZbtGxAahoLNXouM8ca/N7pM2DlJ17XUXsOmFclzZM90h1Rnz46sGUQGW4h3EDbCC410FONB9TMuW0zU5AGi+1I+pZKyvU05yK0mt6mBTShN0khhMigRVIIITJ84+42hoyQW1BW1qWoF+D6NPa7DHdfGzGMhV6VuVwWvt9HKv/EDKMNU6rAhWx2bYkuDu1YryCkZItKxxN3u8SSX9bl7DsqlQahDnMd8ljK6NAlnRRqJ7cDz6PIj+PLk8u5lYM9Vy6dZkqVz1w7DvNxxt3m39rzwTkZyvycK6nC/YClwui3gdw+nOvcSfE4qoy7HamMe0XP1QLCj/o2X/Kr4/JnMN93T9vOgzu7NqzpCDoNrPZteBAz0P2toLQhl6g7umL3NUAn1KqZq4Bur1vfpzJy/cZKVFzyPjgo1bhVLwH+vRBCiBPRIimEEBm0SAohRIatNUkOk0BNkrvEcagKhhzUNYVbENy+AUMMWIMMpJ1hyEjZ5EMzKkov85heRt8NFDKx2N2DH26hQ3GYCBxn4BRO+i2GC/H5MiWHSaDexamW9Fs8jWKuTQTdX+xSWdCeB067hPYFU12Rjonvb4GaJKcl0r5Qz5wZp6J2FAXcE9bQ2a5Aby9ZIz+GXdK7bWsT6i5I++PwoRw8V1Bnxc6JfzkO3U98FmaeI04JRF2VqzDy+QWTWpjvbloVdh5VO6CFLkgXrez5NTWE0c3M7ePQm6QQQmTQIimEEBm0SAohRIatNcmq2aF/gdaR5Oezje0sqzofszbZPkCKGLcNJS0F9Z1JqhlR7+wZG9MLe1IlQ20v07K07TrnKEnHwja0PlJrVPotnmMxo0lyvKoDnYdb38Zp4GT62YxswxpWAZqrp21doJROaPXK8aeT46B9ObBZk+QUR0wn5JaxTLU4OUavpntfkUaHGuWimk9R3CNNsof4xZ7iJFnrxxTIuVbGBaWOYvnCSZk8uuE1pJaGyM+9hWNzxw7mFR1jcCc/v1Uxp0naeVQtk7ZdNfa3u3v23E+BXVXX/l6oN0khhMigRVIIITJs725z6A4W5/F5d9u4YHMhM+zKGRebKr+wywEuBYdPMJwe1g0pxIAr8XBVIK5UNAenl/UQ1sSnwKl4RaYCM8NhTx782ZHSMLlCMwY+hZnULXaD0f0uXX3iNv5tP864WFRRaPS5ECBytyFVklMLmXpp3W085prc1qrm8DaoHjUTLnPcWBFS83pH6XX0qOBc4evKLCjEbagwFdaeE7v5HkLHmiY/Tl3R/S2h2lBJqb+n7Jyr67R9scxX7frW77Dudr1Mx9ws7PFz5jM+rjOK1bHoTVIIITJokRRCiAxaJIUQIoOPE2FPCCHEX6E3SSGEyKBFUgghMmiRFEKIDFvHSf7Ak59K/7J9WiJ2NeQyW//iIx809o/e9Dz72+Lkcl9Dd3IJ+5JSzf6nO99l7Btf8HPG3qyO4PPKbFvs2H3t7J2clvjJd7xl8m9PvPklxu6hfBh3bYyTrpTpenH7ht+lc/rh57zI2APIzVxmn7vcxRFK0lH5tt//6IeN/YSn/SNjN9j6gu7vQB3ysO1AS9v+5//x48b+b5/0FPtbuFYDd0uk/+89pAh6Shf84sfuNPbffaa9btjxr+JUWLo/FcQr7uzY2MSPvem1jrnpDXcYe71OaZqrzdps49JrDcRkFnRcH3njrcb+6dvebOwR7mnX2es+9PZ+LyAueEFxonf+0s8b++bXvdEec5XOYWfPjnPmnJ2Dp84me+eU/e6zn2Dn9sf+8BZjY3xmTbGrQ0sdSDdp+zjYefPjN9jrdBx6kxRCiAxaJIUQIoMWSSGEyLB9+wYuNwWa0KSNAuWVos42zrV0nOiZUEqedLXInSSxFehMvS8uJYW53qz9ca5rMVN6azIW/R611aLkFhSktWFZq7mWstxiA0uykQbpezsOSJLOz+RUT64H5AWXlG/Ned7GptYOzLRsHtxf+u7A7YYxd7vKT/N6aa/bci+V0SvpmkfS8wp8DrhU3TFw+bNQpXMq6boH1l1NN978XODWzLirCPq7c86No9Xgqybte7nM527vnbHXo2pSjvWps/b+Xv8o++yfuR7b3Obn3KMeY7cX0CYWPzvn3BF1wT3cpN/GfIr4sehNUgghMmiRFEKIDFu7267gMB8o4UVuRkHdBQf05SYluizsRhhvM9pjYFcVnTB2mRl2ZbCjY7PcpW1UWTy75+MGs8eCY7GMwTIAXi3r2EwpuEYUejcDXQ9PVaPxcszcI+6GifdspOs6sc2gM/eIrkVZgHzA3+Wq1+D2e74uREWuaQOhPNNugCRbgA/Mx3ssdOA1dppkWYaVKZRP5srH0+YAVeqXS7vjJZUa2zt1BJ8vZ4e5/tsv2H2dSiFeu6dov6ftQe3sQhhPlXfri9Heo/YoXav20F63w4v2Cd2/mOZzt7n2LGy9SQohRAYtkkIIkUGLpBBCZNhekySNA3WoQCEWBYduQLhJnBHWuBWEHZPDPEjHALkhFHnNhsdB3ZFDXKYhTfB5m0pz1J0ONb2GwnZq0s9aCAuZC83BbnrO2bAP35GmTKmHGJrlx5nwKQ6pAd0uctgK26hDzlw61rbN/aXvsvYZ4DqGGU2yoXYcmJbI4V4chubw/sR5tZpPGffPulw/cippsv2MMu5Jcy5C+m1D3QWr2tp7Z0GTPHOQHefco228zd6ZNO5ix96lhtIHqzLds2LM36PY2e2bK2lf++ftfvcv2NCj/YvpmPrNNf9FQW+SQgiRQ4ukEEJk0CIphBAZttckSYqJoIkMlKo1kRVBHildPh7Kk2TQd6D5TFrXWp0CN48zulpHKXGoQwbSwlh2HMdri7ViTRNjOD3FHHLsIH63nBHxamppGkHD7Sl1dKJ3mXytufRHiov1eIz2uveTawej+Pz5FKSxYmwrl5Rj7bOCUnkVtVdlllTirIHryDHAPLkjnNCwsW1Pj2MgfXqEuMthtKX/PE08LGFXhXws68LbVMMGWr02C3uczY61MW5ywbowUfZ0bffT+fVHNBe4Ra7L/K3i26x531fsF472e/hMJRRtxTkzU0J17e+FepMUQogMWiSFECLDN+5ugysweuu69hSeUIY0TOFnhiQ3Gd1CdlurktxtcCnHPh9rxO52Be52TWEfXNW779Nv/SRBboqn1DZMX+Nz8uQWFyaOJz9Ww6EuME5PIVGb3rpYfkzHMZfyNnG3I34+uXKNc6aQjwsz145DmlCKYHebd7WAMJ7F7p7LsbO0lecxDIsrK7EUg9KLdXCPp2fJCNMcW+tus7xSwU/rKj+/2d1eFmnfOws7znKXngU45Wqm4lXZ22vnN+mekQrnNr0dp+3Tc9W2dD8fb817yd3ebJLdbuj4SYZbwDoRyvnnldGbpBBCZNAiKYQQGbRICiFEBh+3yqsTQohvTvQmKYQQGbRICiFEBi2SQgiRYes4yae++KftP0C8l+fYI4rJKzx2NrPr8odvf5+xf+oVzzd2ByGKMVIJM8fpg5ASNVip9VPv+qfGfuItLzN2DfFwVWNjv/pJnGSK0eJUyd/65Tc65id+/vXGLiF+saTyWNy+wWPZNtr2gVe80NjPvOPdk7H/iqG3sXH92sbRDV3azh0tP/FLbzD2U159mz1GuD6htzGyQ39yua9IKZifeucdxn7yy15lbHP+XLqP4kvrRUqXaxb2ft79yhcb+znvvMvYFcRNcpwkx27ilVod2rJi733WUx1z0zs/YOyxTfGqsbWxq9Vg71kN93BZ2rjBX77NzrFfuOO1xm4aiO0tKUa2snGGBQRkUiiye9lP2jn2jo+91P4WnlGOL15TvuBmSON29Ly+8Xn2Or3qPc80Nj6TPMea0t6zHbA5TvvVN7/dzaE3SSGEyKBFUgghMmiRFEKIDFtrko96DLUwgDJXkaufBatLechv5bawzLlHWq2l79PO+9but2+tFgGy2qRlLFNV3K4U8su5jSgJYFgpba51rXPTtrGoQ7KWFictKtJvuXTY3DgDaKd8TgW1UR2hBWs/0yaiIJ3ObeDCk/ZZcOk3PL0iP/1KKv3mQSDz1E44UMm5skq/DVX+uvFvTWsS1te5tB2UPqsX+XGcc67esfroCHNhqGlOrqxOV4KmV/uZUml0mCX8XeBoZa/dmrrG4lQpZqb3hfvtcQQo4dZFq32u46Gx24g6eJ7L6/2TN04KBJz8N5FypjzfcehNUgghMmiRFEKIDFu723uPILcQgx88V4m2vzVd5GY6yp263rprcUiH2K0opODAfncDblLX5yugL2obAuPBTQhUJjlEe+4luFgh5MdxzrmSfBYMAfIc8kN2qDNVsomQq6JN5zANn4IK8DNl5ipyt423Q1Xb49CTbbSK7DgsH3i4Fp7kEu7Yib+91uuGYViBKolPXDtwFFlaOI6KOokOMDc8yzLRzvdqTO5rMeM2crk7vL8dyVTrFbvM8HlmnKOVdalDmcbpnd22iTYEqPXw/M5UMOvGTLlxOsSC9L92TNd1mGvReQx6kxRCiAxaJIUQIoMWSSGEyLC1Jlnv0XpqNATWGSkECHs/+Nbl2DlLISSgy3RH9ruFs3oX6jRdn1//dxYXjT30kE7VW20oOO62l2wMLziJgvShAlo2FBTmwuE1EcJkYpEfK5DeFSBMhjvvsZCDmlWc0STLmkKAMjLPsKFOdpCKNqdDFdxlErRDT+Eyntoq4LWYaLUEhxphywJOOx0pBc5WGpzXu0oKIQpwjoGexop6ppQuXbsw07Fz8Cdfd+6U2Y/U7RT16ZmujB01rSggNGkINt0xRmvjuhDnNFZqEYNtM+h0XNeTXu3S/Z9/WqfoTVIIITJokRRCiAxaJIUQIsPWmmTZnLyeBscxeFaTHB2kH8WZGLya20OmfQWKsezWdl+oF/kiL3jVDaVIgVoxUowl646hAE2ymr+EHMOHGhmX3mJNcgDhbpxJtQzZ9rT5+NQIwk4xM05J+h/GvnrS5TwJRhFzR2fGKYpMLCS3OmVNEraHmbaonsbBtESOi+RWxahJcqrkcQSKmfUQv8qxjUXJ5wTaap+fdwPdbnw2eopdHegeGR1y5nkdIsU1o85Ivx0j65swb/xcYuLJWjBn0XKJtg7Ob4jX/l6oN0khhMigRVIIITJs7W6PVFkjgBsY6A/rBcV2bKAyTJd/e3ftkX13xmLk5CVM3M8I4RVzwRhU5Nx5SJcLFf2aK79Uu+m79d7MSM6FpQ0h8liVhl1XTm2DUIc4U50nUliIzajj/w8pBMhsn4nN8Se7hZPqSxQyM5bgnhV5NzjkqrRzONQkPRRTMvOzga9rb6J6OM3y5HuwTYEZT9fWjE3e6NhRqA7afT40p6cK8RHskebJSOeIcpkP+ZMaOc0Yx2HZjaULh1JFdpjJtcVUYb69cbRzfew9bMuPcxx6kxRCiAxaJIUQIoMWSSGEyOBjnBFshBDimxi9SQohRAYtkkIIkUGLpBBCZNg6TvL1n3yBsQtYX2tnS1GVVD79aH0hfd7YEmVvetYnjX3b3U8ydl2nGKeBYjU3G7bT5763Uus7X2LHueWOJxt7gH0NLaWO1X/N2s1j0ufqjNn2vpc8xzE3v+duY2OaHKfeFZS2NkBcHqdbvee5TzP2C+78mB0HYy65xQYdI8a7DZ1NDX3fzT9j7Oe8+y5j4xlUJHF3RwfGbvdT1zsu/fbB17/G2M96y7uM7THelONLKaYSU0m5xcav3vTfG/sFd9u5YbIDWbKnmEospcbpp29/up3Lzjn3c7/5abs7jCukuEi/sh0C/dGVq5/L/orZ9tZbX2vsV7/5540dh1Rn8GhjWyGsOttmwZeQ3lvZY3rvL/ymsZ//hqfY30L5Qk5BHsiOUFYtlPY6v/81v2fsm2//IbuvAdcF6pzZ2Y6UoYV5M9p79P63fNjNoTdJIYTIoEVSCCEyaJEUQogMW2uSDdWWxxJP5UAlrajUWNmn8l9Vz60Q6IBGWzrMg47B3T0LyqmuYM2f6/TaUEkylDB7LjNF/5dE/LLP59A655zvT86N5XJS/Ui5z6CJhbmMdEpMHTJ5xtMyZfHYj8ePc/KmSSMPlvRwGO49PNmX3Y55z5zLO2lPgV+YSdiNVCoM245OSr9NfgzaGGm5x9Gurf5ncrCpDFvVW+2wdskuQ36sOtBciCefE+f859q1TqB76OHvEdxigsdBc+QeDES3pt/Cc8IlDeLkAcZ20dlhjkVvkkIIkUGLpBBCZNi+WyK721COKNCf1X1n7bJPLnQ9424Xow0n8lDNONC7ckGVoCMcYzHjb7O77fHdn9zjnkov2fCi+dpLfuCwEayqzGER9rcB4lFCmClhRi4Ll8Qyx8Al2bCy2EwnPsZeDeouGNnG312ju43hNvTdSZmyuP09YlcPO0fyONz50ozSz1+3dmVd6ABhXYHc7TCQux2TXc90Ha0DV+cGd3tS/o2Ou7gW6YXuETwrnmQ4LkyOKkhWGnLOkfJArro9yJLOL8AzFrfoaMnoTVIIITJokRRCiAxaJIUQIsP27Rvoz+rY7sC1pKW0pAnAd2uf1wor2j6CthipA2IInA5ZwXfz45RLG2qEHeN60ob4/GKLnRbn/5+Jgw3XiKAHjqSRTCOA8B9mNDyKhTC2Z+3I/tZsvkYdCrvgTVoh0LVEe24YTsMs0OYQJs/HBG0vZroyjtQXxJwDa8STzqDYGmCL3gDcmRA1d7oppbfHhTpk5fKaZEma5FikfReFvfKe7AA2S9dMRRptDdea9b+OI40yYUkTKIwLRw0cG8gaNFzXb6QupN4khRAigxZJIYTIoEVSCCEybK1JDlzGCcuWsTzS2u+i1lLMiBwlaYk9xD5G+m30lMKIJdsmOoWlIk2yH9JJeG7X2ZOmOKTyXzFuoUmO9gKN0M+W22xyyBrqsHNxhUMuTnIi4ZEuNaPbme+SPUKMG+t7HV07a8+0K6XfYkwhzwUWz7D0G19jhnXTANpinMRFcvojbN9Ck/SsSbqT9b+CAgurCJrkTJzkRJPE9FZKWWS7gHjkYmaFqEn6X4JezXGufBsw/XOY7SJzsibJc9fTd6O5FoqTFEKIhxUtkkIIkWFrdzsMJ7vbXFmj4DAWhyEi+WofXBkmeqhATa548DYEKEJIkJ8JNSprCh9q0kmUHYWxUOEW79M/xJlQDOecGx27WIlJs0pOGYOYoDgjIYyU2pVrhBnppnn4/3LO8eb9oh1pnrD73aMLPeP5jJN9QcVs8k3Z5UK3eC7LkuekOR+KyeIQLXMvt0nnnKQEprEHSlEdRzvx4rhK4xb5eVdQdSos5L6zoDC7wlZ5D2WaRxwexJxa2N/uuLRvvvcFpW0WoC2VM9dul8ZxMA5Laz5SmjR+d3Z2T9GbpBBCZNAiKYQQGbRICiFEBh9zwpUQQnyTozdJIYTIoEVSCCEyaJEUQogMW8dJvu3Dzzd2gHjGkto3hMHGYbVQe33T2divV774g8Z+813PMnY8BaXSGhsr5Qsb6+iKk+Mkb/2RN9lxfvc2Y2+O0nGtj+wxri7ZeK/VRYhtc6fNtrt+4S2OefYdb7PHXaXfTOJGObUN4gE5FvDOlz7HjvO299GucF8Un8epXGB7ijv7wEufa8d5+512nD7F7EVKJVxdvmztSxfTOKW9n//sXW819o/f9k+MXe6euvq5KGnOkW3z1uymD7/42ca+6Z132X1hqqAjJilw8JnU/ffROM4597y327Fim56NuLHz7nQ8b+wz7oGrn3dqOydffuvdxn7nO+zcCJA/2NKfIbjvYoR0SI7xffULPmTsd/yqPcdTEHPZU9znfmd7MBxBAHJH333Dz/2WHfeOG42Nz7fnbqYcT2xse/9+6Wd/w82hN0khhMigRVIIITJokRRCiAxba5JuQ+XCwLfnUlSss3Vd+m3bsQJi6VmSg9xtX9jyZqFkjRK+6/OnVlZWzxwgd7smbYnbWbY1aDYzrTCdO6ZtLJTm4vaegUvPgznXhmDSPgBLftE4nIOMWttcyf5cvvkkf5mOCedGHPMD8fngvrlVxUTL9cd+PGEcLn8G21jLnZx7+rjNG8dEg4b8Zj9QrrZbkw01A7g/K+Gp9UNZpKMrSb+NBZWZg/k6DvmzOrNjWzOfqZPdUe72SM8RVH5z9Zh/XveapbHt3xyoTgMfcriGyXAMepMUQogMWiSFECLD1u726sqh/QesQDzy67q1serwEPPu6TDaV+eiTG5xQS6yD1QSCcooo5t+HAWVh6rB5JCCfmFdrHYHOisezbvbQ2v9jCGmcyyo/Bu7Pqat4Zy7zeXs0A1mSSQbEjQzDrvuI44z8pfpmHCU/Dh8H0wIlOewD67Pd7LUwEyOGffN3nVGAphULT+GkTpnNkVyoZulnSc7VIavisl9Laq8ux0KqraOoWT03IRA7mrmOjN7C+sGn4YwrU1rz/Wwtdduhddypqp7MS74X9IhclnEXJiW3G0hhHh40SIphBAZtEgKIUSGb1iTHEGT7Abr6PcUX4LpZ36m/dpA3Qcx9TCUrElyO4dkh5AfhzVJX0MIDOmE3dLqJfUm6T1tPqLJOedcpBiiHqWm2oY1OWorYWW7a9Qk7VZjjePJmiR3UmQmlfZR82NtaRLiBOPMapKsLZ2cosngsJPQKP7uJAQoo+WyJmlCwOb16TjaCVNVaTLsVXaeNNFqkjWE5pRFfizWJFG3K+j+FqQ7YsRMmNHwlo2dv7s7u2nMYI+/DPb8AlyLuXtUjPxc4PNNGiv9Nmwvtx+L3iSFECKDFkkhhMigRVIIITJsrUle3reaZDTOPcX6UUpgCZoHl7iaQFpTAA2zID0zcCtJo0nm1/+C+t6inllQrOKSUq8iyj1DPl7NOed2SdMsML2MW6FOdpeOJc78nzZJ1TPpdQy3lN1erJnogahn0nUvSNQqi+0Fr0mIHmhp00OgWF13LXGSpNdiLOfkp1xiLn0uZmKAnXOuotJjdZN0uXrXangNpTA2fRqsivlrV5UnxwqOPbf5Je0Qzyk7inOrlf3tJX+QtrX2+NcdtQjGazszkK9IR4WlgP/8EP3JMcHbxLIyepMUQogMWiSFECLD1u72pSsHxkZ3tqAK4WVFYS3gYgdHf8pnyF3DlKmC06km7jamXuXdkUDbMZyI3cloM6+cxzAlqtxyHDvkbjtwdzh8qutpbPBDIqdfEZEqEtkMTzpfTrfDTLS53K1sVXO7raD7WcJ1j3P3iA8yk182e8zXALrfXKWJwbMrtggBqqg6D7rbzSlyt0nKaTbQDWCmOk9FchKGOW3I7e0GO26EezZJkyWO1nb+467b3l67DbnfQ0SZZuY61yRZQamusrHnOlBIV9dj5Sm520II8bCiRVIIITJokRRCiAw+zsVHCCHENzF6kxRCiAxaJIUQIoMWSSGEyLB1nOQzbvoeY/tMqiGnDxZQ4r1sbIrfr7z5U8Z+9bufZ+xzj/2Wq58XeztmG5edx5S4QHFiN9/wMmO//w/eZn+L3QIp/rKnsvPdJtlH+zZO7JU3vsoxt334tcY+vJxi444Obcm29ZG9diPcopFaUvz6G37e2M94/VuN7U2Mm70ek4g+jHulNhkffsUtxr7pbe+343Qpzi50Nuau279k7SsXrn6OVK7uN95l78nTXvuP7Ti7p9M4Vb5z5jhgxz8bn/fRVz7f2E9/y3vsvjDmktNOe2sHyFGtg42BfP9r7P1xzrkXveWNxj59/X76fN0Vs21ntHNrBzqNlp29gz/9jI8Z++4PPM3YQ5fub9vba9UN1MqkTM9VRc/2s573LmN/8M4XGxunGYVjuk1H8YsQ1xsLW0Lu5S+0c+yt732mPcZldexn55wbKT20h3vIf4J56U/ZcY5Db5JCCJFBi6QQQmTQIimEEBm21iR9PLn0UqSSTgPl3I6btBb3Qz4ss99QPwSUeSLlLpON5b7mys5z7jbaBZdrq6n0G+Yjz5Sscs65nTNW48MWuy3pnY5ajmJ+8zhTT2rkVgloT3Kb5+wMk/YNJ2/zpA95LNk/9180zStT4n9SlowP6uR88gn00xHGjZ3VGceN1VwLOJ9yMV82z1HudoQJPmm3HLi9Mmjug90PE+k5G1vQaKltMwvUNl87P+f6nvR6mL9DZB2clhuc2zPh2jxOiTUFA9dwsN8NBd4X5W4LIcTDihZJIYTIsLW7XVMoAFb47elVeejZhQTXoMu7Pv2G3Bsst8SFt7ncGZZKm+n4xx5YMCFAtF8qFxXg/5aqmb+Eza51t1sI36iuWPc6BJIuoIzVMORLcQ2tdQUdhPJwlTWuiI5yynx3QXIrMdymI7lg5PPD+5v/PzqSS4mur+cTorAtDPWYOx926z10JXRURixQaTzv0vkVft7dLkvqGAhdD3meMaYppcuPNZC7jban6x4oFCtkOhEy7LoP4G5PqpLR6eFzxq45E2k7FlffbFimoDXESBxyt4UQ4mFFi6QQQmTQIimEEBm21iQXFAaDJdIHCpPoOtaAMC1oRpOkEu8j6psscnDHPEhFnHTaIybd9k7e7QQMXcGwjJOoG6tJLpYRttnzLUurgbUQutEd5VtFDGurd5m2CnSrubsg6l1zoTmRwpTGDro/tkd2HEqtC0U6nzhzk7irX9xcg55pDmJGn47ULTDi+XFrA9IUfdpeV/OaZF2v7FjQBZBTK1krHcCe60LAoXYoZ4fKzoWqtPMzmDYpc/0SaV5F0MHpjwgcijViy5CZcXy04/QwrXpKFY0FzRu4R24L3ZjRm6QQQmTQIimEEBm0SAohRIatNcmmtnpJDyJHy/F7lBaErU5Hiptj+s5qWP1mBdtsmbVQc6zcnH4CX82m5WUES+ecB8WLS7IdR0UaUN2k69Es7TE3thqc66DeVHD5a+cj6X+QtulJxBpZeMQ2onOtUakcWgSBKPaki0ayQ/rt7JUjPXPs0r44Dnak+DdfQFzgzKtAEUkn9GmcUNpjiKRRFlDiq17Mt5StF3Z/JTxXBWmDHAqJ8Z4c58qMfNIYB0ylDMvKPtsR5sZknhAxsrYN8bbctpjmVfSZfNaZcXrQ6ntOX/X2Ho0Br7k0SSGEeFjRIimEEBm2drcrSl3y8Kf0hsJYRkqtw8rA/GrMjG7f2Ierh65+DitKpWvsGl9AmMD/uxaQdhx2G2zFkvmROH2yqtNx75y213XobcVtbyqY5K/d3i6l0EGa2zDasIg1pYeOI6R0ljPTYm1dxgIlEkrjcxN3+wC2UcgLM9q5YKo+zYT1BAjHKcv8davCJWtXqUJ4Eez5cFgLXqrF7rzcU+/a39dNug8lV/TnEBq4h77Kj1U1S2NHeB8K5F5zOuQ4QHWeGe90oJRPnGc9h4rR+aBEEmdSMkeS9HoIO9ywREcyzQA2Vy3fBr1JCiFEBi2SQgiRQYukEEJk8JHbhwkhhLiK3iSFECKDFkkhhMigRVIIITJsHSf5mlf+Q2Nj2abVYGPh2O6h5H032m2/9qtfNvZzXvN4Y+8+8rqrn0898lvMtjOP+Hb73dPnrn4uKYzsOd/zUmN/6I/eYewAKY2B0htZtsVuh9zl7Vl/+4WOueuL7zY2lnbaUPmz9ZG9PvvnD4797Jxz73j5e4z94ttp7DHFw23WNi7y4NDabZf+v6ypZNdH/+mbjH3TK28zth9TWl8YbIpf9A8ZewwPXv3cU+n/X/vl3zP2T77kqcYeYrq/vjhltoXS5nPWTYrRaxobI/r+22439i1vea6xy8WF9LmmMmoUz1dCyuliaWNcf+EnPuqY2z/9LGNXze6xn51zrupbsuE6H9l4v5ue/F5jf+gjt9iBW0hRLexxxmDTIXsIb6QMVHfLLXYuvP3tLzV21x6mz3T8PbXjwDPwFLv5i7febezX32Gv23qdrsVqbcvzdXTQHZzQSHGdd73tc24OvUkKIUQGLZJCCJFBi6QQQmTYWpPsNlbD6qA8/FFrczSPuFxWnXQ33+RbEPjK/jYG0Bs8tfOkkmym/NlM+CdriVj+LFI+Nu8K04b9TDuKyXE554qQ9KG6sfqQLZ1v23s2pHkx1z16z9g9tJhdHVISbmmvXQt58YXP57c2Nd8HKH8W7FzoBtalYC7MXLt6ccX+A7aYiFbf4jYLDZRKW9Qz59NQ+bMFtlSgegHBvleYvOdiPnc7+pPbaFCKv8m9d865ALUJ5kr0FYXVaD3WXiisBsnH1DvU8PLl+QZqndCCHthOtMGTy9v5If+8bqg1SdtiCUWrSXL/kRo113Dt74V6kxRCiAxaJIUQIsPW7na7Jhveso/W1vU5aK071pxJNrtqEyobQuIChgaQm8elpOA1O86UFeOwnphxt13GLZx3tqf/E6HLFhob+lA1do/NMlVjP3XOhogw1z3abt+s0n0pL1PJL0+l08A9DVxpnmgqug8QJuMrKo9FUkzcwG/HmXGWl41dRCi5t6FOgtRZsQF3dFnn79KiIZeygfs/KeHF7nZysWPY4nEK5JLjXKBq44FcQ5Riwkw5u6K0pdL8AFJNoErkLAG4FMYTx/zzymE9bZeu5YbufUeXGT1sP9NpEkN+/nJfycXue+tul96eewXSQ+G3XvKuojdJIYTIoEVSCCEyaJEUQogMWzvoXIod7Z7CAPqeStyDdjiGmdCcQDoV2qSjuUg5UxgGMheak93OemV28yz8ddRDR9JOx8itI1r4Lp8v7ZevD0ankNZZ2Wgh2xlhzJ9gfZbL8EMXPzofT3MBb7+fuZAFzYUSwot8YXWogjTmcidN7aLJt4koajoO1Pv4NSKyTpguXMl643FjFRziBfeFQ6+o1YCHi+fnNHdq1zEOcNzcOdOxvgtzbiYEaORUQxAa48htRimcysR05fXpSH0kCkwjLm0X1SqQDSFPHGK3DXqTFEKIDFokhRAigxZJIYTIsH1aIqeXQfkhbBnrnHMDaR4DaA/DjCYZaXs0OYB2nEgaXYwppitEm3rFcKqg3S//A//L9umPfwlreAmS7FxPsYMRys5FKkPGDCNdD/jsKyrxtUvnD7GBfuac6uvs9n6djrnfkAZJV7MAnWouxpQVvgJaw4aSAncrq42VddKlQpM/n1BRiirGEdJrBF8aTDFlvfE4StLEsN3wJLaX5nsAzXJOzx1pLgwQ+xqjvbIj6awDpBWP3CJ4Mg5p6mBG0teDpxhTf7I2z0RaU/A68jUtqRRciZqk4iSFEOLhRYukEEJk2PrdM9Kf6LGKziTFj7P68LPfxj0FzPf5txw2gK/sc+Ows5dzofm713YO02+ffO24crKDVLxp5Rv6aiaMgjwd50tOgdveDaZC1jYyK++d2QpKMyN5z24wuJsluWcU5RMgzc0X+fCSybUxoSocxjL5dRqTd3QME5cTPnM6LLvU1p6RrejZwBCayfPJz6/5bt4N5p1F+7CbbZ7LHF0Tdhy8jiWlb7L7jZIIp3pug94khRAigxZJIYTIoEVSCCEy+MiChBBCiKvoTVIIITJokRRCiAxaJIUQIoMWSSGEyKBFUgghMmiRFEKIDFokhRAigxZJIYTIoEVSCCEy/N9WsqAbINxULwAAAABJRU5ErkJggg==\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "plt.figure(figsize=(4, 4))\n",
        "image = x_train[np.random.choice(range(x_train.shape[0]))]\n",
        "plt.imshow(image.astype(\"uint8\"))\n",
        "plt.axis(\"off\")\n",
        "\n",
        "resized_image = tf.image.resize(\n",
        "    tf.convert_to_tensor([image]), size=(image_size, image_size)\n",
        ")\n",
        "patches = Patches(patch_size)(resized_image)\n",
        "print(f\"Image size: {image_size} X {image_size}\")\n",
        "print(f\"Patch size: {patch_size} X {patch_size}\")\n",
        "print(f\"Patches per image: {patches.shape[1]}\")\n",
        "print(f\"Elements per patch: {patches.shape[-1]}\")\n",
        "\n",
        "n = int(np.sqrt(patches.shape[1]))\n",
        "plt.figure(figsize=(4, 4))\n",
        "for i, patch in enumerate(patches[0]):\n",
        "    ax = plt.subplot(n, n, i + 1)\n",
        "    patch_img = tf.reshape(patch, (patch_size, patch_size, 3))\n",
        "    plt.imshow(patch_img.numpy().astype(\"uint8\"))\n",
        "    plt.axis(\"off\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vrjIAlUnbPsG"
      },
      "source": [
        "## Implement the patch encoding layer\n",
        "\n",
        "The `PatchEncoder` layer will linearly transform a patch by projecting it into a\n",
        "vector of size `projection_dim`. In addition, it adds a learnable position\n",
        "embedding to the projected vector."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GDMUnmwrbPsH"
      },
      "outputs": [],
      "source": [
        "\n",
        "class PatchEncoder(layers.Layer):\n",
        "    def __init__(self, num_patches, projection_dim):\n",
        "        super().__init__()\n",
        "        self.num_patches = num_patches\n",
        "        self.projection = layers.Dense(units=projection_dim)\n",
        "        self.position_embedding = layers.Embedding(\n",
        "            input_dim=num_patches, output_dim=projection_dim\n",
        "        )\n",
        "\n",
        "\n",
        "    def call(self, patch):\n",
        "        positions = tf.range(start=0, limit=self.num_patches, delta=1)\n",
        "        encoded = self.projection(patch) + self.position_embedding(positions)\n",
        "        positions.shape\n",
        "        encoded.shape\n",
        "        return encoded\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MiGiSta6bPsH"
      },
      "source": [
        "## Build the ViT model\n",
        "\n",
        "The ViT model consists of multiple Transformer blocks,\n",
        "which use the `layers.MultiHeadAttention` layer as a self-attention mechanism\n",
        "applied to the sequence of patches. The Transformer blocks produce a\n",
        "`[batch_size, num_patches, projection_dim]` tensor, which is processed via an\n",
        "classifier head with softmax to produce the final class probabilities output.\n",
        "\n",
        "Unlike the technique described in the [paper](https://arxiv.org/abs/2010.11929),\n",
        "which prepends a learnable embedding to the sequence of encoded patches to serve\n",
        "as the image representation, all the outputs of the final Transformer block are\n",
        "reshaped with `layers.Flatten()` and used as the image\n",
        "representation input to the classifier head.\n",
        "Note that the `layers.GlobalAveragePooling1D` layer\n",
        "could also be used instead to aggregate the outputs of the Transformer block,\n",
        "especially when the number of patches and the projection dimensions are large."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iW6HMEdUbPsH"
      },
      "outputs": [],
      "source": [
        "\n",
        "def create_vit_classifier():\n",
        "    inputs = layers.Input(shape=input_shape)\n",
        "    # Augment data.\n",
        "    augmented = data_augmentation(inputs)\n",
        "    # Create patches.\n",
        "    patches = Patches(patch_size)(augmented)\n",
        "    # Encode patches.\n",
        "    encoded_patches = PatchEncoder(num_patches, projection_dim)(patches)\n",
        "\n",
        "    # Create multiple layers of the Transformer block.\n",
        "    for _ in range(transformer_layers):\n",
        "        # Layer normalization 1.\n",
        "        x1 = layers.LayerNormalization(epsilon=1e-6)(encoded_patches)\n",
        "        # Create a multi-head attention layer.\n",
        "        attention_output = layers.MultiHeadAttention(\n",
        "            num_heads=num_heads, key_dim=projection_dim, dropout=0.1\n",
        "        )(x1, x1)\n",
        "        # Skip connection 1.\n",
        "        x2 = layers.Add()([attention_output, encoded_patches])\n",
        "        # Layer normalization 2.\n",
        "        x3 = layers.LayerNormalization(epsilon=1e-6)(x2)\n",
        "        # MLP.\n",
        "        x3 = mlp(x3, hidden_units=transformer_units, dropout_rate=0.1)\n",
        "        # Skip connection 2.\n",
        "        encoded_patches = layers.Add()([x3, x2])\n",
        "\n",
        "    # Create a [batch_size, projection_dim] tensor.\n",
        "    representation = layers.LayerNormalization(epsilon=1e-6)(encoded_patches)\n",
        "    representation = layers.Flatten()(representation)\n",
        "    representation = layers.Dropout(0.5)(representation)\n",
        "    # Add MLP.\n",
        "    features = mlp(representation, hidden_units=mlp_head_units, dropout_rate=0.5)\n",
        "    # Classify outputs.\n",
        "    logits = layers.Dense(num_classes)(features)\n",
        "    # Create the Keras model.\n",
        "    model = keras.Model(inputs=inputs, outputs=logits)\n",
        "    return model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SKeeRzEibPsI"
      },
      "source": [
        "## Compile, train, and evaluate the mode"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LT5yq4brbPsI",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "ff0db5a0-af58-4103-8d8e-dd3766507cb9"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1/6\n",
            "176/176 [==============================] - 83s 414ms/step - loss: 4.4782 - accuracy: 0.0455 - top-5-accuracy: 0.1587 - val_loss: 4.0246 - val_accuracy: 0.0924 - val_top-5-accuracy: 0.2820\n",
            "Epoch 2/6\n",
            "176/176 [==============================] - 71s 402ms/step - loss: 3.9472 - accuracy: 0.0956 - top-5-accuracy: 0.2870 - val_loss: 3.5611 - val_accuracy: 0.1632 - val_top-5-accuracy: 0.4052\n",
            "Epoch 3/6\n",
            "176/176 [==============================] - 67s 382ms/step - loss: 3.6943 - accuracy: 0.1306 - top-5-accuracy: 0.3649 - val_loss: 3.3959 - val_accuracy: 0.1822 - val_top-5-accuracy: 0.4464\n",
            "Epoch 4/6\n",
            "176/176 [==============================] - 67s 382ms/step - loss: 3.5167 - accuracy: 0.1577 - top-5-accuracy: 0.4207 - val_loss: 3.1916 - val_accuracy: 0.2260 - val_top-5-accuracy: 0.5068\n",
            "Epoch 5/6\n",
            "176/176 [==============================] - 67s 382ms/step - loss: 3.3535 - accuracy: 0.1887 - top-5-accuracy: 0.4665 - val_loss: 3.0561 - val_accuracy: 0.2458 - val_top-5-accuracy: 0.5448\n",
            "Epoch 6/6\n",
            "176/176 [==============================] - 67s 380ms/step - loss: 3.2261 - accuracy: 0.2148 - top-5-accuracy: 0.4959 - val_loss: 2.9366 - val_accuracy: 0.2740 - val_top-5-accuracy: 0.5676\n",
            "313/313 [==============================] - 7s 24ms/step - loss: 2.8977 - accuracy: 0.2816 - top-5-accuracy: 0.5803\n",
            "Test accuracy: 28.16%\n",
            "Test top 5 accuracy: 58.03%\n"
          ]
        }
      ],
      "source": [
        "\n",
        "def run_experiment(model):\n",
        "    optimizer = tfa.optimizers.AdamW(\n",
        "        learning_rate=learning_rate, weight_decay=weight_decay\n",
        "    )\n",
        "\n",
        "    model.compile(\n",
        "        optimizer=optimizer,\n",
        "        loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "        metrics=[\n",
        "            keras.metrics.SparseCategoricalAccuracy(name=\"accuracy\"),\n",
        "            keras.metrics.SparseTopKCategoricalAccuracy(5, name=\"top-5-accuracy\"),\n",
        "        ],\n",
        "    )\n",
        "\n",
        "    checkpoint_filepath = \"/tmp/checkpoint\"\n",
        "    checkpoint_callback = keras.callbacks.ModelCheckpoint(\n",
        "        checkpoint_filepath,\n",
        "        monitor=\"val_accuracy\",\n",
        "        save_best_only=True,\n",
        "        save_weights_only=True,\n",
        "    )\n",
        "\n",
        "    history = model.fit(\n",
        "        x=x_train,\n",
        "        y=y_train,\n",
        "        batch_size=batch_size,\n",
        "        #epochs=num_epochs,\n",
        "        epochs=6,\n",
        "        validation_split=0.1,\n",
        "        callbacks=[checkpoint_callback],\n",
        "    )\n",
        "\n",
        "    model.load_weights(checkpoint_filepath)\n",
        "    _, accuracy, top_5_accuracy = model.evaluate(x_test, y_test)\n",
        "    print(f\"Test accuracy: {round(accuracy * 100, 2)}%\")\n",
        "    print(f\"Test top 5 accuracy: {round(top_5_accuracy * 100, 2)}%\")\n",
        "\n",
        "    return history\n",
        "\n",
        "\n",
        "vit_classifier = create_vit_classifier()\n",
        "history = run_experiment(vit_classifier)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TATDw8JabPsI"
      },
      "source": [
        "After 100 epochs, the ViT model achieves around 55% accuracy and\n",
        "82% top-5 accuracy on the test data. These are not competitive results on the CIFAR-100 dataset,\n",
        "as a ResNet50V2 trained from scratch on the same data can achieve 67% accuracy.\n",
        "\n",
        "Note that the state of the art results reported in the\n",
        "[paper](https://arxiv.org/abs/2010.11929) are achieved by pre-training the ViT model using\n",
        "the JFT-300M dataset, then fine-tuning it on the target dataset. To improve the model quality\n",
        "without pre-training, you can try to train the model for more epochs, use a larger number of\n",
        "Transformer layers, resize the input images, change the patch size, or increase the projection dimensions.\n",
        "Besides, as mentioned in the paper, the quality of the model is affected not only by architecture choices,\n",
        "but also by parameters such as the learning rate schedule, optimizer, weight decay, etc.\n",
        "In practice, it's recommended to fine-tune a ViT model\n",
        "that was pre-trained using a large, high-resolution dataset."
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": [],
      "include_colab_link": true
    },
    "environment": {
      "name": "tf2-gpu.2-4.m61",
      "type": "gcloud",
      "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-4:m61"
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.9"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}